Day 11: Embeddings and Vector Representations — How Machines Map Meaning

Day 11: Embeddings and Vector Representations — How Machines Map Meaning

To enable an AI to work with your company's data, you must convert that data into measurable form. Words are unquantifiable; numbers are. The intermediary is the embedding. Visualize a colossal map with each sentence in your company assigned a pin. Sentences sharing a topic are clustered together, while unrelated ones are spread apart. An embedding pin's coordinates is the pin's location.

An embedding model transforms a text snippet into a dense vector, typically a list of 768 or 1536 numbers. You never read these numbers directly; their relationships matter. Similar meanings produce similar vectors, effectively placing them close together on the map. The term "network latency" and "packet drop" are near each other, whereas "financial forecast" is far removed. The same holds true for a question you pose; it too is converted into a vector and lands next to the text that answers it.

The key to embeddings lies in the distance measure known as cosine similarity. It compares the direction of two vectors. A score approaching 1 indicates the vectors point in the same direction, meaning the texts have similar meanings. A score close to 0 signifies the vectors are unrelated. In essence, embedding search surpasses keyword search in relevance.

If you ask, "Why is the site slow?", a keyword search would miss a document discussing high latency on the web tier. However, an embedding search would locate that paragraph because the meaning is synonymous, even if the words differ.