The missing piece of agent self-improvement

Opik is an open-source platform for observing and evaluating AI agents, offering a comprehensive solution for developers building LLM-powered applications. The platform covers the entire LLM application lifecycle, from the initial development stage to production monitoring. Opik is built by Comet and is licensed under Apache-2.0, allowing users to self-host the entire platform for free.

This means that the server backend, web application, tracing, datasets, experiments, evaluations, prompt management, and agent optimization components are all included within the repository.

Opik caters to several user groups, including ML engineers developing LLM-powered agents, AI teams transitioning from prototypes to production, and engineering teams that require open-source, self-hostable observability. The platform's open-source nature is significant as it enables self-hosting, ensuring that all data remains within the user's environment without any external data transfer or enterprise sales discussions.

Users can start logging traces by wrapping functions with the @track decorator, capturing multi-step traces that include LLM calls, tool executions, retrieval steps, and other agent activities.

The platform supports various features such as datasets, experiments, code-based metrics, LLM-as-a-judge evaluation, and online evaluation. However, it's worth noting that Opik is framework-agnostic, supporting its SDK, OpenTelemetry, and framework-specific integrations. This allows for seamless integration into different workflows, with SDKs available for Python, TypeScript, and Ruby.

In terms of deployment, Opik can be run locally or in a user's own infrastructure, using Docker for local setups or Kubernetes for scalability.

Opik also offers a suite of client libraries and a REST API to interact with the Opik server, including SDKs for Python, TypeScript, and Ruby. The Python SDK allows for easy configuration using the opik configure command, which prompts for the Opik server address or API key and workspace for Comet.com. The platform also supports a wide array of frameworks, including Google ADK, Autogen, AG2, and Flowise AI, with support for custom integrations as well.

For evaluating LLM applications, Opik provides metrics such as LLM as a judge metrics and pre-built heuristic metrics, along with the ability to create custom metrics. These evaluation tools can be used during development through datasets and experiments, with evaluations even being run as part of a CI/CD pipeline using the PyTest integration.