Your OpenAI Evals freeze on Saturday. Here's how to get them out, and keep testing on any model
OpenAI stops hosting its Evals service. As of Saturday, October 31st, your evals become read-only. As of Monday, November 30th, the Evals Dashboard and Evals API become inaccessible. Also on that day, the Agent Builder and reusable prompts (the v1/prompts API) are discontinued. If you created test suites in Evals, the former date marks when you will lose the ability to change your evals, and the…
OpenAI is discontinuing its Evals service on October 31st, 2023. After the shutdown date, the Evals Dashboard and API will no longer be accessible. If you have existing evals, you will only be able to view them until November 30th, 2023. To preserve your evals and their last run, you can migrate to Promptfoo, OpenAI's open-source evals framework.
Alternatively, you can export your eval results and build tests using the Evals API. You can use the LLMShift tool for this purpose, which is freely available under the MIT license. LLMShift allows you to pull evaluation results and generate test cases using Python 3.9+ and your OpenAI API key. You can run the tests on any model, including OpenAI and non-OpenAI models.
Additionally, you can integrate LLMShift into your CI workflow to automatically check for differences between your evals and the new model.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
