Small Specialized Models Are Eating the AI Stack (While Everyone Watches Frontier LLMs)

Every few weeks, headlines announce a new frontier model with promises of revolutionizing AI. However, most AI agents actually rely on smaller, specialized models for the bulk of their work. Embedding, reranking, entity extraction, and OCR are the tasks that consume the most inference resources, and these are best handled by smaller models.

Superlinked's analysis shows that open-source models have nearly the same quality as hosted frontier models on standard text embedding benchmarks, while costing significantly less. For example, embedding a billion-token corpus can cost between $10 to $17 for self-hosted models, compared to $120 to $130 for hosted APIs. Similarly, generation tasks like Qwen3.6-27B match GPT-5.1's performance despite being released months earlier.

However, small models don't scale as easily as large models due to infrastructure challenges. Serving many small models requires separate GPU pools for each, leading to wasted resources if not managed properly. Enter SIE, an open-source inference engine that packs multiple models onto a single GPU, addressing this issue. This allows for efficient and cost-effective inference for a variety of tasks, from search and retrieval to document OCR.

While large models like frontier LLMs still have their place in novel reasoning, small specialized models provide excellent performance and affordability for the majority of agent tasks.