Embeddings Cannot Say No: An Intent Detector's Real Numbers
My message detector worked on my test set. On messages it had never seen, it missed one real case out of two. TL;DR : I built a small detector that spots messages asking for an action. It relies on embeddings, a technique that turns a sentence into numbers. It works well, except for one thing: it does not understand negation. "The outage is fixed, thanks" fires exactly like "outage". I explain…
The system the reporter built attempts to identify messages requesting action using embeddings, which convert sentences into numerical lists. While the detector works well on known messages, it struggles with negation, failing to distinguish between real actions and harmless statements like "The outage is fixed, thanks." The reporter explains that a fixed similarity threshold doesn't work, as scores are tightly packed and raise false alarms.
Instead, the reporter suggests comparing messages to neutral anchors, determining if they're more similar to a tool or a neutral phrase. This approach results in fewer false positives but higher false negatives. The reporter points out that embeddings can't inherently recognize negation, as both the positive and neutral sentences share the same topic.
To address this limitation, the system relies on the larger language model (LLM) in the second stage to interpret negation and select the correct action. The reporter concludes that the detector is a gate, not a judge, with high specificity (catching most real cases) but lower sensitivity (allowing some harmless messages through).
The final configuration mentions using a small multilingual model and a low threshold to catch a wide range of real cases, accepting some noise in exchange for overall accuracy.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
