Building a Hybrid AI Agent With Local and Cloud Models

In this article, the author demonstrates how to build a practical AI software system using a 3B-parameter quantized LLM model running on a Raspberry Pi 5. The AI agent manages various tasks such as email triage, code generation, service management, and infrastructure monitoring. The agent utilizes a combination of local and cloud AI models to optimize performance and efficiency.

The author explains the rationale behind the local-cloud split, which allocates about 70% of the agent's work to the local 3B model and the remaining 30% to a cloud API. This split is based on the strengths of each model: the local model excels at procedural, deterministic tasks with high success rates, while the cloud model handles more complex tasks requiring judgment or creativity.

The author argues that using the cloud model for everything would be costly and result in higher latency due to the time required for cloud API requests.

The author outlines the architecture of the AI agent, which consists of a user request, a task planner driven by the cloud model, step execution by the local model, verification of results with deterministic code, and a loop that repeats until all tasks are completed or a failure occurs. This architecture improves reliability by preventing the local model from managing complex plans and by ensuring that the cloud model handles only its strengths, such as planning and executing commands.

To address potential failure modes, the author recommends deterministic system state checks, reiterating goals at each step, and using tolerant parsers that can handle format inconsistencies. The author has successfully built various real-world software projects using this approach, including a domain availability checker with an API backend, a monitoring system, and more.

In conclusion, the author presents a practical engineering decision to decentralize AI compute by running a significant portion of an AI agent on an affordable $80 computer. The local-cloud split, along with the architecture and failure mode mitigation strategies discussed in the article, demonstrate that this approach can effectively build real software that ships, serves requests, and earns money.