A /models list with no modality field, and five entries that can't chat
If you build a model picker by reading GET /models and dropping every id into a chat completion, you will ship a picker where a third of the options are broken. I tried it against Groq. The endpoint advertises 14 models. Sending each one the same minimal chat request: answered a minimal request : 9/14 failed outright : 5/14 The five failures are not outages. They are models that were never chat…
When constructing a model picker by reading the GET /models endpoint and adding every ID into a chat completion, it is crucial to understand that a third of the options will likely be broken. To illustrate this, I tested this approach against Groq, an endpoint that advertises 14 models. However, out of those 14, only 9 were able to successfully respond to a minimal request.
Specifically, 5 models outright failed, each responding with a 400 error. These failed models were not experiencing outages but were found to be non-chat models.
The five non-chat models that failed are whisper-large-v3 400, whisper-large-v3-turbo 400, canopylabs/orpheus-v1-english 400, canopylabs/orpheus-arabic-saudi 400, and groq/compound 429. These failures were not due to issues with the servers, but rather because these models were not capable of chat interactions.
Breaking down these failures, we find two speech-to-text models (whisper-large-v3 and whisper-large-v3-turbo) and two text-to-speech models (canopylabs/orpheus-v1-english and canopylabs/orpheus-arabic-saudi). Additionally, there is a router model (groq/compound) that was rate limited at the time of the test. The only way to differentiate between chat models and non-chat models is by recognizing the names, as there is no field indicating the modality of each model in the list.
In addition to these findings, the test also revealed that guard models (llama-prompt-guard-2-22m and allam-2-7b) answer chat requests but reject a max_tokens value of 8192. The error message states that max_tokens must be less than or equal to 512 for the guards and 4096 for allam. This discrepancy in output ceilings is another aspect that cannot be easily discovered from the model list response.
Lastly, groq/compound was identified to be a router model. The error message for the 429 response indicated the underlying model as openai/gpt-oss-120b, which is the only place where this routing information is visible from the outside. Despite this, none of this information is documented in the model list response. Therefore, if one were to programmatically enumerate models and probe each one, it would be essential to cache the actual responses received, rather than relying solely on the catalogue. This practice would ensure a more accurate and functional model picker.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
