I raced six models against each other on DigitalOcean Inference. The cheapest one won.
Every time I put a model behind an endpoint I make the same lazy decision. I pick whatever I used last time, or whatever I read about most recently, and I tell myself I'll benchmark it properly later, and later never arrives because there is always something with an actual deadline on it and comparing model latencies feels like procrastination even when it isn't. I never do it. Not once. So I…
For six different models, the author conducted a benchmark on DigitalOcean's Inference platform. Instead of selecting models based on prior experience or recent reading, he aimed to benchmark them properly. To do this, he created a Python script that simultaneously sent the same prompt to six models and measured each model's response time and cost per run.
The author found that three unexpected things occurred during this process. First, the integration with DigitalOcean's inference endpoint was extremely simple, as the endpoint spoke OpenAI's API. Second, despite using six concurrent streams, the server remained synchronous and didn't require any asynchronous programming or orchestration.
The most significant discovery was that all six requests executed successfully, even though the first table showed varying response times for each model. This led the author to realize that the catalog of available models was inaccurate, as some models that appeared available were actually not accessible to his subscription tier. The author's default selection had included a model that the author's account couldn't use.
The author realized that most of the information returned by the /v1/models endpoint was unreachable, and he should have assumed this when building a menu from that endpoint. He also noted that a leaked model access key could lead to unnecessary spending, while a leaked API token could lead to losing the entire account.
In summary, the author discovered that DigitalOcean's inference endpoint was easy to integrate, six concurrent streams didn't require complex programming, and the available models were not accurately represented in the catalog. The author also learned about the importance of using the correct credentials and the potential consequences of not doing so.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
