Euro open source specialist proposes LLM gateway to ease token bills

Open-source database service provider Aiven has reduced its AI token costs by 80% by implementing open-weight models and developing a gateway to simplify the process of swapping between models as needed. The Finland-based company, which specializes in hosting open-source databases, has recently expanded into machine learning. Aiven's VP of strategy, Ville Lehto, explained that while it's challenging to create an LLM infrastructure that works across different cloud providers, their goal was to manage their own token costs and maintain control over their AI usage.

Open-weight models, which have publicly released learned parameters and can be run locally or hosted at a lower cost than Western frontier models, have gained popularity among developers seeking to reduce their costs. In response to demand, Aiven built an internal gateway to help developers work more easily with open-weight models.

The gateway, built on the open-source framework LiteLLM, offers budgeting, access management, and other components to facilitate the use of open-weight models. Aiven continues to use frontier models, primarily for planning purposes, but relies heavily on open-weight models for their actual workloads. The company plans to offer the gateway as a service to its customers, helping them manage token costs and swap open-weight models more easily within their workflows.

Notable Aiven customers include German ecommerce company Conrad, UK payment processing company Dojo, cyber security vendor Sophos, and communications firm Avaya.