I Built an Observability Tool Because I Was Tired of Debugging LLM Apps By Hand

The author created Obyflow, an open-source observability tool for AI applications, after finding debugging LLM applications difficult and time-consuming. Traditional debugging methods, like stack traces, don't provide enough information about issues such as retriever failures or model misbehavior. Obyflow addresses these problems by collecting detailed telemetry on LLM calls, embeddings, vector databases, and tool calls.

The tool is designed to be local-first, using SQLite with no backend setup required, and it allows users to choose any LLM to interpret the raw evidence and provide plain-English explanations for incidents.

The project is structured as a pnpm + Turborepo monorepo using TypeScript, with a separate Python package that mirrors the event contract. There are ten event types, including trace, log, metric, error, embedding, vector_op, chain, tool_call, llm_call, and custom. These events carry specific information, such as provider, model, prompt/completion token counts, latency, similarity scores, and query latency.

The diagnosis layer examines correlated events for failure patterns, like failed steps, timeouts, zero documents returned, and significant duration regressions. Diagnoses are scored based on evidence volume, statistical deviation, anomalous metrics, deployment correlation, and whether real parent/child span relationships are established.

The LLM is used to generate diagnoses from the evidence, with a validation process ensuring the model cites real evidence and not hallucinated information.