Kolibri: A Sovereign Open-Weight Model

On the day of German reunification, the release of a new English-German mixture-of-experts transformer model, Kolibri, marked a significant milestone. Kolibri boasts an impressive 78 billion total parameters, with 3 billion active parameters, and supports context lengths of up to one million tokens. The model is open-source, downloadable with full weights on Hugging Face, and can be used under the Apache 2.0 license terms.

Kolibri is the culmination of continuous iterations in the model training process. Initially, a model training pipeline was built and validated by creating Kolibri Origin, a 30 billion total, 3 billion active parameter model with a 65k token context window. The same pipeline was then used to create Kolibri, which enabled running hundreds of ablation experiments and maintaining stable pre-training without the need for human intervention in cases of hardware failures or dropped data connections.

Regular monitoring of training metrics and custom benchmarks ensured the model's performance remained optimal.

Kolibri is a specialized language model designed for sovereign, mission-critical work in regulated areas, including public administration, industrials, and aerospace. It has been optimized for German, focusing on reasoning, math, agentic behavior, and other capabilities crucial for production. This specialization aimed to enhance performance in specific use cases for customers, enabling them to monitor the economic impact of Kolibri and ensure a growing return on investment (ROI) over time.

One of the key aspects of Kolibri's development is its sovereignty, which combines two dimensions: the model's construction and its transferability to customers. The model offers full supply-chain integrity, accounting for every decision made during data ingestion, pre- and post-training, and final evaluations. Transparency is a core principle, with customers having full freedom of deployment and intellectual-property safety, inheriting compliance as a natural property of the model.

Kolibri's small and efficient size allows customers to run it efficiently on-premise, without sending internal data to third-party inference services. The model strikes a balance between model capability and deployment costs, achieving the best quality versus serving cost trade-off on the Pareto frontier for both English and German.

Compared to other models with similar capabilities, Kolibri matches models with up to four times its active parameter count, such as Nemotron 3 Super, in terms of quality across various tasks, including math, coding, grounding, and long-context tasks.

To ensure Kolibri's performance meets the specific needs of customers in various sectors, the model was developed with a focus on domain-specific language, regulatory, and procedural realities. A dedicated evaluation suite, tailored to the skills, workflows, and edge cases required in sectors such as the German public sector, aviation, manufacturing, and the automotive industry, was created.

These evaluations were performed using synthetic training environments, allowing Kolibri to improve without ever training on customer data.

Kolibri was trained with abstention data and the Merlin-Arthur protocol, enabling it to confidently say "I don't know" when the answer is not present in the context. This feature was highly valued by customers, and Kolibri's performance in tracking and validating abstention accuracy was continuously monitored. The model's bilingual German/English design, with 21.3% of pre-training tokens being German, was achieved through the use of a native German and English tokenizer and a sparing use of translation (6% overall) to include organic German data throughout the training process.