Teaching a Computer Word Meaning Without a Dictionary
When I started learning Natural Language Processing (NLP), one question bugged me: how does a computer understand words? The honest answer is that it doesn't. Computers only understand numbers. So before a model can work with language, we have to turn words into numbers, and the way we do it makes a huge difference. That is what word embeddings are about. In this post I'll explain what they are,…
When the author began studying Natural Language Processing (NLP), they pondered how a computer comprehends words. Computers only understand numbers, so before a model can work with language, it must convert words into numbers. This is where word embeddings come into play. In this article, the author will clarify what word embeddings are, demonstrate a few simple examples that can be executed in seconds using plain Python, and explore how Word2Vec learns these embeddings.
A word embedding is a word represented as a list of numbers, or a vector, such as "cat" becoming [0.21, 0.85, -0.13, 0.42, ...]. These numbers are not typed in manually; instead, they are derived from text. The key concept is that words used in similar contexts receive similar numbers, allowing a machine-learning algorithm to treat them as related rather than completely distinct.
This is why word embeddings are significant: the model can leverage its knowledge about "cat" when it encounters "dog." The traditional method, one-hot encoding, assigns each word a vector with a single 1 and the rest 0s, resulting in unrelated words like "cat" and "dog" having no similarity. This method also has a drawback: using a 100,000-word vocabulary, every vector would have 100,000 slots, most of which are zeros.
Word embeddings address both issues by employing short, dense vectors (typically 50 to 300 numbers) where distance conveys meaning. The importance of context cannot be overstated. In a demonstration, the author collected the words that appear in the same sentence as each word, computing the intersection of these sets to show the relationships between words.
For instance, "cat" and "dog" share three context words, while "cat" and "car" share only one. By analyzing these neighboring words, it becomes evident that cats are more closely related to dogs than to cars. Word2Vec achieves this same idea but on a larger scale. It slides a small window over text and learns from the words nearby, employing two approaches: Skip-gram, which predicts the surrounding words given a center word, and CBOW (Continuous Bag of Words), which predicts the center word given its neighbors.
As the model improves at these predictions, its internal vectors adjust, resulting in words appearing in similar contexts having similar vectors. With millions of sentences, well-known outcomes like king − man + woman ≈ queen may emerge. Building your own embedding using plain Python is possible. Each word becomes a vector of counts, representing how frequently every other word appears in a sentence with it.
Then, cosine similarity is used to compare the vectors (1 indicates the same direction, 0 means no commonality).
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.