Running an AI Agent Locally: ADK, Gemma 4, and Docker Model Runner
This article was originally published on Medium (Google Cloud Community) . Cloud LLMs are great for production. But during development, every API call has latency, costs money, and requires credentials. What if the LLM ran on your machine, right next to your agent? In my previous article , I built a football statistics agent using Google ADK , BigQuery MCP via Cloud API Registry, and Gemini 2.5…
Running LLMs on Your Machine: Local Deployment with ADK, Gemma 4, and Docker Model Runner
In this article, the author explores the process of deploying a football statistics agent locally using the Google AI Toolkit (ADK), BigQuery MCP via Cloud API Registry, and Gemma 4 running via Docker Model Runner. The core idea is to eliminate the need for cloud inference, eliminating costs, and API keys. The full source code is available on GitHub.
What is Docker Model Runner?
Docker Model Runner is a feature within Docker Desktop that allows users to pull and run large language models (LLMs) locally. It functions as an OpenAI-compatible API, enabling any tool designed for OpenAI's protocol to utilize it.
To enable this feature, users must activate TCP access on Docker Desktop. For macOS users, this step is necessary as the model will be accessible at http://localhost:12434/engines/v1. The model can then be pulled using the command `docker model pull ai/gemma4:E4B`. It is available at the same endpoint as the OpenAI API, eliminating the need for an API key.
What is Gemma 4?
Gemma 4 is Google's latest open-weight model family, offering several variants with different on-disk sizes and parameters. The E4B variant, currently the default, is a balanced choice offering a good trade-off between quality and speed for local development. Unlike its predecessors, Gemma 4 does not support native function calling, which can cause issues when integrating with an agent framework.
The Agent Code: A Single Model String Change
The only change required in the agent code to switch from Gemini to Gemma is a modification of the model string. The original code uses `gemini-2.5-flash`, while the local version uses `ai/gemma4:E4B`. This change allows the agent to communicate with Gemma 4 via Docker Model Runner.
Function Calling Issues
However, running the agent with Gemma presents a challenge. Gemma lacks native function calling support, causing the agent to crash when attempting to execute functions. To overcome this, the author suggests converting tool calls into text prompts that Gemma can understand, then extracting function calls from the model's response. This workaround allows the agent to function properly with Gemma 4.
In summary, running an AI agent locally using ADK, Gemma 4, and Docker Model Runner is a viable option for developers looking to bypass cloud inference costs. While there are challenges to overcome, such as functioning without native function calling support, the benefits of local deployment, including faster inference and no cloud dependencies, make this approach worth considering.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.