An easy way to try local AI with braindead-local-intelligence

A new project aims to make it easier for computer users to experiment with local AI. The project, named braindead local intelligence, simplifies the process of running both language models for chat and image generation models on a single computer. The creators of the project likened the setup to starting a simple AI system without much prior setup, hence the name "braindead local intelligence."

The project utilizes containers to run several services on a user's computer. These services include a locally hosted LLM server that is compatible with the OpenAI API, an image generation server, web search capabilities provided by SearXNG, and a web interface called Open WebUI. All these services are accessible through a single address and can be secured with a token if necessary.

A key feature of the project is its adaptive model loading system. Only one model is loaded into memory at a time; the model is unloaded after a few minutes of inactivity, freeing up GPU memory. This allows the system to efficiently manage GPU resources, even when multiple models are available for use.

The project also offers a variety of models suited for different tasks. These include small text models, models capable of understanding images, image generation models, and models that can modify existing images. Switching between models is a straightforward process of updating the configuration and restarting the system. The user's computer only needs a Linux system with podman 5.0 or newer, running in rootless mode.

The project is currently optimized for AMD graphics cards but is compatible with NVIDIA cards as well, though it has not been tested with NVIDIA hardware.

To run the project, a user needs a GPU with a certain amount of VRAM and sufficient RAM. The project was tested with models up to 12GB of VRAM. The project documentation provides step-by-step instructions for installation and setup. The simplest way to use the system is through the Open WebUI web interface, accessible at http://127.0.0.1:3000. The system also includes a web search feature, a chat functionality, and the ability to upload images for generation or modification.

The creators have tested the system with a 35B model that performs well even with web search, which helps the model to understand and deliver accurate responses. While the system does not match the capabilities of large AI services, it provides a valuable opportunity for users to explore local AI and understand what their hardware can achieve.