Model Cascade: making LLM classification cheaper
Many LLM workloads are classification tasks. This can get expensive, and I believe it is going to become more and more important, especially with the proliferation of software factories. So what is Model Cascade ? In short, it is a way to make a deterministic system around a cheap model and make it give us the same results as the expensive model. Core concepts A Proxy is the cheap model. It…
Model Cascade is a technique to make low-cost language models act similarly to expensive ones. It uses a "proxy" cheap model and an "oracle" expensive model. The proxy provides an output and confidence score while the oracle gives its output and whether the proxy was accurate. Two targeting modes are used: matching the oracle on a target percentage of records or meeting precision and recall targets with a fixed oracle call budget.
The key idea is to sample the oracle's response, run the proxy model multiple times, and determine a threshold where the proxy's confidence matches the oracle's labels. This allows using the cheap proxy model for most records while only calling the expensive oracle for a smaller subset. The BARGAIN paper shows up to 86% cost reduction on various datasets by applying this principle.
To implement it, pick an oracle and proxy model, confirm the proxy can provide useful confidence scores, write the task prompt, run BARGAIN_A to determine the fraction of records the proxy can handle, and adjust the prompt or proxy if necessary. Additional optimizations include rewriting prompts, using relevant document chunks, and searching for the cheapest cascade sequence. The BARGAIN library supports swapping providers and has dependencies like numpy and pandas.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
