Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard

Coding agents like Claude Code and OpenAI Codex CLI execute tasks that involve reading files, editing code, and running tests until the work is completed. The payment is based on the task, not individual model calls. The cost of a task depends on the number of turns, history replayed per turn, and retries or failures. Routing coding agent requests to the appropriate model is crucial; thus, measuring cost per solved task is recommended.

NVIDIA NeMo Switchyard is an open-source library that aids in model selection for each request based on defined models and routing strategies. LiteLLM integration supports two strategies: Stage routing and Random routing. Stage routing analyzes recent tool activity and results to choose between two candidate models, while Random routing selects from configured models with optional weighting for traffic splits and A/B comparisons.

LiteLLM supports deploying Switchyard routing with the AWS Cloud Development Kit (AWS CDK) in a non-production AWS account. The solution architecture involves LiteLLM and the Switchyard plugin running in a container image on Amazon Elastic Container Service (Amazon ECS) in private subnets. Requests flow through various steps, including validation of gateway key, filtering by AWS WAF, and forwarding to the ECS task.

LiteLLM calls the selected model through the Amazon Bedrock Converse or ConverseStream API, and the response is returned through LiteLLM and the load balancer. Logs and metrics are collected for monitoring and analysis.