Google Launched the Gemini Agent. Let's Build a Tiny Model Router in TypeScript.

The Problem

A router must answer three questions for each task: which models can handle it, which of those is the cheapest, and whether there is still budget to run it. Most code only answers the first question by hardcoding a model name. That works great for capability but leaves price and budget concerns unanswered.

Step 1: Describe the Models

A model consists of four facts: its tier (small, medium, or large), price per 1,000 tokens, maximum context size, and whether it supports tools. A task describes its required tier, token count, and optional tool support. An override field allows a human to override the system's choice. Google plans to let users select a model themselves initially.

Step 2: Decide What Fits

A model is considered a fit if its tier rank is higher than or equal to the task's required tier, the task's token count does not exceed the model's max context size, and the task requires tools only if the model also supports tools. This ensures models are neither too small nor too large for the task.

Step 3: Pick the Cheapest Fit

After filtering out unfit models, the remaining models are sorted by price. The cheapest model that still meets the task's requirements is selected as the route. For example, a small model may be chosen over a medium or large model if it meets the task's needs at a lower cost.

The Bigger Idea

The core concept is that a router should make data-driven routing decisions based on model capabilities, cost, and budget constraints, rather than relying on hardcoded model choices or blindly prioritizing larger models. A tiny TypeScript implementation demonstrates the idea, routing seven example tasks through various states like running, pausing, rejection, and resuming. Note that the example models, prices, and tasks are fictional and not reflective of Google's actual Gemini agent implementation.