I Built My Own J.A.R.V.I.S. — A Local, Privacy-First Multi-Agent AI Assistant
I wanted a personal AI assistant. Not a chatbot. Not a copilot. Something that actually does things — controls my desktop, reads my screen, talks to my smart home, answers questions in Finnish or English, and never, ever sends my data to a server I don't control. The cloud options are good. They're not mine . So I started building one. This post is the architecture document for that project. It's…
The author wanted a personal AI assistant that could control their desktop, read their screen, communicate with smart home devices, answer questions in both Finnish and English, while keeping their data private and never sending it to external servers. To build this, they started by using Ollama, a lightweight local LLM, as the brain for the system.
Ollama is configured with function calling, which allows it to return a structured JSON decision on what actions to take instead of executing raw commands directly. This keeps the system safe and prevents unauthorized actions like deleting files. The author built four main agents within Docker containers, each handling a specific task - controlling the Linux desktop, providing visual input and analysis, managing Home Assistant smart home devices, and fetching external data when needed.
Each agent communicates with Ollama using a RESTful API and can be individually controlled and updated. Safety features include human-in-the-loop confirmation for sensitive operations and a strict function allowlist that limits the models to only authorized actions. The entire system is kept on the local network, providing full privacy and control over the AI assistant.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
