How LLMs Actually Work: A Practical Guide for Product Managers

For Product Managers developing AI products, understanding Large Language Models (LLMs) is crucial. However, you don't need to be an ML researcher. Instead, focus on grasping the core concepts. At its simplest, an LLM is a machine learning model trained on vast datasets to recognize language patterns and generate outputs based on input. A practical way to conceptualize an LLM is as follows: "An LLM takes tokens as input and predicts what token should follow next."

The process begins with tokenization. When you input text, the model doesn't receive it directly. Rather, the text is transformed into tokens, which aren't always complete words. For instance, the phrase "internationalization" may be broken down into smaller tokens. This distinction matters because tokens influence things like API costs, context window usage, latency, and throughput.

Following tokenization, the tokens are converted into numerical representations known as embeddings. Neural networks process numbers, not text. Embeddings act as numerical representations that enable the model to understand relationships between information pieces.

The truly transformative part, however, is the Transformer architecture, introduced in the 2017 paper "Attention Is All You Need." Transformers rely on attention mechanisms instead of recurrent models. A simplified Transformer block consists of self-attention and feed-forward networks. Self-attention allows the model to understand relationships between tokens, with the concept of Queries, Keys, and Values facilitating this process.

Understanding these fundamental aspects of LLMs equips Product Managers to make informed decisions regarding AI product development, such as choosing between using pre-existing LLM APIs or building custom models, tackling performance issues, or deciding on techniques like Retrieval Augmented Generation (RAG) or fine-tuning.