Everyone’s Optimizing Prompts. Nobody’s Optimizing the Data Going Into Them

The conversation surrounding reducing AI expenses has centered on prompt crafting, while less focus has been placed on the data returned by these tools. Developers concentrate on developing prompt libraries, tallying tokens, and striving for concise messages, yet the data pulled from APIs by agents is seldom discussed. This oversight is significant.

The issue: What is happening? While prompt tuning can feel like genuine effort, it may seem as if there's little you can do to declutter the response you receive. This is because the data often appears beyond your control. The internet is flooded with advice on prompt engineering. The structure of the incoming data is often overlooked.

However, the bill doesn't differentiate between a well-directed prompt and the accompanying JSON data. It's worth noting that many assume this aspect is immutable. Enter SerpApi's MarkDown feature. The company has introduced a new method of delivering results, highlighting this disparity. By conducting a search for "coffee," SerpApi was able to contrast the costs.

In full JSON format, the response cost 24,723 tokens. Using MarkDown results reduced this to 6,435 tokens, a 74% reduction, without any changes to the prompt. Further filtration brought the JSON down to 8,486 tokens, and MarkDown to 1,298 tokens. The work remained the same; only the data format changed. These numbers, from SerpApi's own comparison, could potentially save up to 50% of the average cost across the company's 100+ APIs, with some endpoints seeing an improvement of up to 90%.

The improvement isn't magical; it's a result of SerpApi's MarkDown feature removing unnecessary information that language models don't need, such as tracking and redirect links, icons, metadata, and duplicate fields. Despite the removal of these elements, the model still receives all the necessary titles, snippets, sources, and links to function effectively.

However, not all APIs benefit from MarkDown. If your code requires specific types or strict JSON schemas, then JSON remains the better option. The crucial question for developers is: what consumes the response, a language model or a parser? For agents, Retrieval-Augmented Generation (RAG) pipelines, and chat tools, MarkDown often serves the same purpose with fewer tokens.

For legacy systems that depend on rigid typing and field validation, JSON is the safer choice. Developers must decide based on the consuming entity, not on an abstract notion of which format is inherently better. To test MarkDown yourself, no new stack is needed. If an API supports it, it's typically just a query parameter, a different route, or an Accept header.

Run the same API call in both JSON and MarkDown formats and compare the token count. If your workflow is LLM-first and you don't rely on exact JSON schemas, the lighter format could likely reduce costs without modifying your prompts. Many APIs already support MarkDown, usually accessible via a query parameter, a different route, or an Accept header.

By comparing a real response's token weight in both formats, you can determine if a cleaner version would still work for your agent. Pulling this lever, which is largely ignored in the industry, could significantly reduce costs.