I Tried to Teach a Computer "Apple" (It Thought It Was a Fruit, an iPhone, and a Vector)

Hello, DEV Community! 🙌 I'm writing my first post here on the topic of how computers understand language. As I explored how models handle text, I realized the surprising truth: computers don't comprehend human words, they only understand numbers. To make a computer grasp concepts like words having multiple meanings, I discovered the power of word embedding techniques.

The first step in turning words into numbers is one-hot encoding. This method assigns a unique binary code to each word in a giant list. For example, the words "cat," "dog," and "apple" would be represented as separate arrays of 1s and 0s. However, this approach has significant drawbacks - it wastes a tremendous amount of memory by creating large arrays filled with unnecessary zeros.

Moreover, computers don't have any inherent understanding of the relationships between words; they treat each word as a completely separate entity.

Researchers like Mikolov et al. at Google introduced a more efficient alternative called Word Embedding. Instead of using one-hot encoding, Word Embedding represents words as points in a high-dimensional space. These points are plotted using dozens to hundreds of invisible dimensions, each representing a hidden concept such as "is_animal," "is_food," "is_tech," or "grammatical_type." For instance, the word "apple" would be plotted near the cluster of fruit-related concepts and also close to the "iPhone" neighborhood.

The computer learns these coordinates by sliding a "context window" across millions of sentences on the internet. It observes which words consistently appear together in similar contexts, allowing it to cluster words with similar meanings in close proximity on the coordinate map. Because the computer processes vast amounts of data, it effectively "reads" the relationships between words and places them accordingly on the vector space.

One of the most fascinating aspects of word embedding is the ability to perform mathematical operations on word coordinates. For example, the famous equation "King - Man + Woman = Queen" demonstrates how you can manipulate word vectors to derive new meanings. By starting at the coordinate for "King," subtracting the "male" direction, and adding the "female" direction, the resulting coordinate lands on the "Queen" word.

This powerful geometric concept showcases the deep semantic connections that word embedding techniques can reveal.

In conclusion, word embedding provides a valuable framework for bridging the gap between machine language and human language. By representing words as high-dimensional points in a geometric space, computers can begin to understand the nuanced relationships and meanings inherent in human words. This innovative approach has opened up exciting possibilities for natural language processing and has become an essential tool in modern AI applications.