Semantic Caching And The Near-Miss Problem: Why A Single Similarity Threshold Is Not Enough

2026-11-11T08:19:32Z

CarmineQ62: Created page with " Semantic caching is a practical way to cut the cost and latency of large-language-model applications. Instead of asking the model to generate a fresh answer every time, the system stores the [https://www.ft.com/search?q=response response] to a previous query and reuses it when a new query means the same thing. To decide whether two queries are "the same," the cache compares their embeddings—vectors that represent meaning—using cosine similarity. If the cosine sim..."

Semantic caching is a practical way to cut the cost and latency of large-language-model applications. Instead of asking the model to generate a fresh answer every time, the system stores the [https://www.ft.com/search?q=response response] to a previous query and reuses it when a new query means the same thing. To decide whether two queries are "the same," the cache compares their embeddings—vectors that represent meaning—using cosine similarity. If the cosine similarity between the new query’s embedding and a stored query’s embedding passes a single similarity threshold, the stored answer is returned. In deployed products, that threshold is typically shipped as a default between 0.80 and 0.95. The logic seems straightforward: high similarity means same intent, low similarity means different intent. But the reality is messier, and the weak point is a class of inputs called near-miss queries. These are two questions that look almost identical—differing by a single word, a small phrase, or word order—yet require completely different answers. A customer asking "How do I cancel my subscription?" and "How do I pause my subscription?" are near-misses. The embeddings will be close, because most of the words and the grammatical structure are shared. The cosine similarity will likely be high. But the correct answer to one is a permanent termination process, and the correct answer to the other is a temporary hold. A semantic cache that returns the stored answer for "cancel" when the user asked about "pause" produces a false cache hit: a confident, fluent, and entirely wrong response. A 2026 preprint by Guerin Green of Novel Cognition examined this problem directly. The author built a labelled set of 243 English customer-support question pairs across eight domains. The set contained 101 heavily reworded paraphrases—queries that use different vocabulary and syntax but share intent—and 102 near-miss pairs that differ by a word or by word order but need different answers. A further 40 lightly reworded paraphrases were reported separately. The pairs were scored by cosine similarity using the open embedding model Qwen3-Embedding-0.6B. The result was counterintuitive. The near-miss pairs scored higher on cosine similarity (median 0.801) than the genuine paraphrases (median 0.691). The area under the ROC curve was 0.144, with a 95% bootstrap interval from 0.097 to 0.195. That means the distributions were not merely overlapping; they were inverted. A high cosine similarity score did not indicate that two queries meant the same thing. It indicated that they were worded similarly, which is precisely the wrong signal for near-misses. The practical consequence shows up at a typical default threshold. At a threshold of 0.85—the default of one commercial service—the embedding model matched none of the 101 genuine paraphrases, meaning it missed every legitimate reuse opportunity. At the same threshold, [https://zenodo.org/records/23667256 click here] it falsely matched 29 of the 102 near-miss pairs, a 28.4 percent false cache hit rate. To get false matches down to zero, the threshold had to be raised to 0.985, and even then, 2 of the 40 near-verbatim repeats were still missed. In other words, a threshold that is safe for near-misses is so strict that it defeats the purpose of caching, and a threshold that is useful for paraphrases is dangerous for near-misses. The preprint also found that cosine similarity correlated strongly with word-set Jaccard overlap (Pearson r = 0.695), including within each label. That suggests the embedding model was largely measuring lexical overlap rather than deeper semantic distinction. Interestingly, adding a query-instruction prefix—a short phrase that tells the model what to do—halved the false matches at 0.85, bringing them down to 12.7 percent, without changing the ordering of the pairs. That is a useful trick, but it does not solve the underlying problem. The author’s core argument is that a similarity threshold is not a transferable constant. It is a property of one embedding model working on one traffic distribution. A threshold that works for Qwen3-Embedding-0.6B on this customer-support dataset may fail on another model or another domain. Other embedding models need their own measurement. The preprint describes a labelled near-miss protocol for setting a threshold in a given deployment, and it releases the pairs, the scores, and an analysis script under CC BY 4.0 so that practitioners can recompute every statistic and test their own data. For engineers building semantic caches, the lesson is not to abandon the idea. It is to treat the threshold as a tunable parameter that must be validated against a realistic set of near-miss queries, not as a universal default. The near-miss problem is not an edge case; it is a structural feature of natural language. A single cosine similarity cutoff will always be a compromise between missing genuine paraphrases and serving wrong answers. The only way to know which side of that compromise you are on is to measure it with the actual queries your system will see.

User:CarmineQ62

2026-11-11T08:19:23Z

CarmineQ62: Created page with "Semantic caching for chatbots: learn to set thresholds that catch true repeats but reject near-miss queries that would produce wrong answers. my homepage ... [https://zenodo.org/records/23667256 click here]"

Semantic caching for chatbots: learn to set thresholds that catch true repeats but reject near-miss queries that would produce wrong answers. my homepage ... [https://zenodo.org/records/23667256 click here]

User blog:CarmineQ62

2026-11-11T07:11:42Z

CarmineQ62: Root blog page created

XIV Lore - User contributions [en]

Semantic Caching And The Near-Miss Problem: Why A Single Similarity Threshold Is Not Enough

User:CarmineQ62

User blog:CarmineQ62