Small Models Have Arrived

In recent weeks, the capabilities of gpt-5.6-luna have captured the attention of tech enthusiasts. This model demonstrates impressive speed and intelligence, with an average of 100 tokens per second. However, the cost associated with using Luna remains a significant concern. Running complex research threads can quickly rack up a large bill, even when the model is searching through thousands of emails, resulting in tens of cents in API costs.

Despite this, the introduction of the new Pareto frontier option for coding tasks has made the expensive and capable models, such as Fable 5 and 5.6 Sol, the go-to choice for many developers. This has led to an apparent lack of consumer AI companies, which can be attributed to the high token costs. Before the advent of AI, consumer apps typically followed a specific playbook, but with the rise of AI, inference costs have increased dramatically, making it challenging for consumer apps to break even.

For instance, building a personalized daily news site that researches and curates content from sources like Twitter and Reddit would cost $1 to get started, while charging $30 per month is not feasible for most consumer apps. However, with Luna, the results are quite promising, with an average cost of just $0.10. This has opened up new possibilities in the world of business.

Peter, a co-founder of Segment, has observed that ~95% of his work falls into two categories: managing calls, nudging people, and performing essential tasks. Despite having an IQ of 180 and employing a highly capable technical mind, Peter still finds himself spending most of his time on these routine tasks. He believes that as demand for frontier-level models continues to grow, particularly in fields requiring novel breakthroughs or discoveries, there will also be a significant increase in demand for fast, cheap, and good-enough models.

These models are poised to cater to the needs of professionals who require quick, efficient, and reliable support, such as coworkers, vendors, and customers. To make fast, cheap, and good-enough models a reality for business, several challenges need to be addressed, including new harnesses, prompt injection safety, roles, and permissions.

Nevertheless, the author remains optimistic and confident that the necessary solutions will be developed.