My CI Doesn't Know Which Free Model It's Calling Anymore

On a Tuesday morning, my GitLab job failed. The error message reported a 404 status code for a missing /v1/chat/completions route. I inspected my codebase, but discovered no recent modifications. The free model route had shifted locations, while my YAML configuration continued referencing the outdated URL. The cause was not the model itself, but rather the way it was wired.

I had inadvertently duplicated the same endpoint across multiple locations. Every time a provider altered a route, I was forced to manually edit YAML files, push changes, and wait for updates. This tedious process amounted to endpoint babysitting, not model engineering. To alleviate this issue, I employed a solution provided by MonkeyCode.

Their service offers access to free model routes, along with a free server option. The sidecar component performs a single task: determining which model should be called by my jobs. Below is the Python code for the sidecar:

```python

import json

import os

from http.server import BaseHTTPRequestHandler, HTTPServer

class Handler(BaseHTTPRequestHandler):

def do_GET(self):

if self.path == "/config":

body = json.dumps({

"base_url": os.environ["MODEL_BASE_URL"],

"model": os.environ["MODEL_NAME"],

"timeout_s": int(os.environ.get("MODEL_TIMEOUT_S", 20)),

}).encode()

self.send_response(200)

self.send_header("Content-Type", "application/json")

self.end_headers()

self.wfile.write(body)

elif self.path == "/healthz":

self.send_response(200)

self.end_headers()

self.wfile.write(b"ok")

else:

self.send_response(404)

self.end_headers()

if __name__ == "__main__":

port = int(os.environ.get("PORT", 8000))

HTTPServer((0.0.0.0, port), Handler).serve_forever()

```

The sidecar's primary responsibility is to provide the current model endpoint. It does not store any tokens or request bodies. Its sole purpose is to act as a config shim, not a proxy. By introducing this sidecar, I simplified the CI process. The job now queries the sidecar for the model URL, rather than hardcoding it within the YAML configuration. The test cases ensure the sidecar's configuration adheres to the expected shape:

```python

import os

import requests

SIDECAR_URL = os.environ["MODEL_SIDECAR_URL"]

def test_config_shape():

r = requests.get(f"{SIDECAR_URL}/config", timeout=5)

r.raise_for_status()

data = r.json()

assert set(data.keys()) == {"base_url", "model", "timeout_s"}

assert data["base_url"].startswith("https://")

assert isinstance(data["model"], str) and data["model"]

assert 1 <= data["timeout_s"] <= 60

```

The actual model-calling script remains unchanged:

```python

import json, os, urllib.request

url = os.environ["MODEL_SIDECAR_URL"]

cfg = json.load(urllib.request.urlopen(f"{url}/config", timeout=5))

```

Adopting this approach over a YAML-based solution provides several advantages. The sidecar ensures a single point of configuration change, reducing the risk of inconsistencies across multiple locations. By separating the configuration logic from the model-calling script, the codebase remains cleaner and more maintainable. Additionally, the sidecar's lightweight nature minimizes the overhead on the CI pipeline, allowing for faster job execution.