How to Configure a Custom LLM Proxy in OpenOPC (and Debug LiteLLM)

To configure a custom LLM proxy in OpenOPC, one must first set the desired model in the .opc/config/llm_config.yaml file. Initially, the model was set as glm/glm-5.2. However, when OpenOPC attempted to run the agents, LiteLLM crashed with an error stating that the LLM provider was not provided. This was because LiteLLM requires the LLM provider to be specified explicitly.

OpenAI has a convention of prefixing model names with "openai/", which LiteLLM didn't recognize when using a custom api_base. To resolve this issue, the model string in the config had to be updated to openai/glm-5.2. This adjustment allowed LiteLLM to format the request correctly. However, another error occurred when LiteLLM stripped off the openai/ prefix before sending the request to the proxy, resulting in the model name glm-5.2 being passed directly to the proxy, which wasn't recognized.

The solution was to modify the config by stacking the OpenAI prefix with the actual model name: llm: default_model: openai/glm/glm-5.2. This configuration ensured that LiteLLM used the OpenAI API format while passing the correct model name glm/glm-5.2 to the proxy. Upon restarting OpenOPC with this configuration, the routing worked as expected.

It's crucial to note that when connecting OpenOPC to a custom proxy, one should query the proxy's models endpoint first to retrieve the literal string it expects. This string should then be prefixed with openai/ in the OpenOPC config. Despite facing a temporary ServiceUnavailableError due to high proxy server capacity, the connection and routing were ultimately successful.

For those managing custom API proxies and model routing in their agent stacks, the key considerations are whether to configure prefixes like this in LiteLLM or to rewrite the requests at the proxy level before they reach the models. Sharing experiences and preferences on this topic can be beneficial for the community.