Reduce LLM Cost for Catalog: Compare Small Models to Summarize, Classify, and Extract JSON

To reduce the cost of using Large Language Models (LLMs) for catalog data processing, the best approach is to measure the cost per accepted record and then route each job based on its difficulty. This involves counting prompt tokens before making a call, using a small model for simple tasks like summarizing, classifying, and extracting JSON, and reserving a stronger model for exceptions.

It's essential to batch work that can wait and focus on preserving the required fields at the lowest cost per accepted result rather than the lowest token rate.

When comparing small models to larger models like GPT-4, it's crucial to use a set of records that includes ordinary catalog text and awkward cases that cause real damage. These cases include missing attributes, conflicting units, embedded markup, multiple products, and descriptions where the correct answer is "unknown." The evaluation set should be frozen and periodically re-run to account for changes in tenant data.

The decision-making process should involve comparing routes by quality-adjusted spend, measuring valid-result rates, field or label accuracy, wrong categories and invented attributes, cost per accepted result, input, output, retries, and review work, tail latency, and review rate. These metrics help connect the model choice to the tenant's bill and show whether interactive work should be routed elsewhere.

Based on the analysis, a small model is usually a candidate for predictable, low-ambiguity records. However, a stronger model may win for descriptions with nested attributes or high business cost of error. The router can use cheap signals like input length, language, field coverage, and previous validation results to make routing decisions, but it should log the rule used to make each decision.