The Promptfoo Acquisition Made Me Realize I Was Evaluating LLMs on Easy Mode

In March 2026, the OpenAI company acquired Promptfoo, a platform widely utilized by 350,000 developers for assessing and red-teaming OpenAI's own models. This acquisition likely did not render Promptfoo obsolete, but it prompted the author to question a crucial aspect of evaluating language models (LLMs): who should the evaluation tool be accountable to?

Most existing LLM evaluation frameworks, such as DeepEval, Ragas, and LangSmith, share a commonality: they route scoring through an LLM API call. Consequently, these pipelines are subject to three uncontrollable dependencies - the tool vendor, the judge model vendor, and any potential pricing modifications by either of them in the coming quarters.

During the initial development of LLM Judge, the author did not focus on vendor independence. Instead, the primary concern was cost. The author was running evaluations on a coding Q&A dataset, and each DeepEval run was consuming LLM API calls outside of the pre-established budget. Consequently, they sought a simpler, more cost-effective approach.

The author opted for a more straightforward method: training a classifier. Utilizing Sentence Transformers to embed the responses and logistic regression for classification, the classifier achieved an 75% accuracy rate on coding Q&A tasks, with each sample evaluation taking approximately 8ms and having a cost of $0 per run. Importantly, this classifier operates entirely offline, providing immediate feedback suitable for continuous integration and delivery (CI/CD) environments.

This approach offers several advantages:

- No API key is required, making it deployable in air-gapped environments or restricted CI systems.

- It is vendor-independent, eliminating the risk of vendor lock-in.

- It provides fast feedback loops, capable of evaluating 1,000 outputs in under 10 seconds.

- It is customizable to specific domains, allowing fine-tuning on domain-specific data and definitions of "good."

However, this method does necessitate labeled training data. Even with as few as 50-100 examples, a useful baseline classifier can be achieved.

To get started, simply install llm-judge with 'pip install llm-judge' and then train the classifier on your data using local evaluations. There are no sign-ups, no API keys, and no monthly bills.

The author's experience with Promptfoo serves as a valuable reminder: the optimal evaluation tool is one that one understands, owns, and can run independently from the model's creator. If your current evaluation setup differs, especially if it does not rely on API-dependent evaluations, please share your approach in the comments.