{
  "id": 8012568,
  "title": "I Built an Observability Tool Because I Was Tired of Debugging LLM Apps By Hand",
  "url": "https://urgent.news/2026/09/17/i-built-an-observability-tool-because-i-was-tired-of-debugging-llm",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-17T06:37:54.000Z",
  "source": {
    "name": "HackerNoon",
    "slug": "hackernoon",
    "url": "https://hackernoon.com/i-built-an-observability-tool-because-i-was-tired-of-debugging-llm-apps-by-hand?source=rss"
  },
  "original_language": "en",
  "account": "The author created Obyflow, an open-source observability tool for AI applications, after finding debugging LLM applications difficult and time-consuming. Traditional debugging methods, like stack traces, don't provide enough information about issues such as retriever failures or model misbehavior. Obyflow addresses these problems by collecting detailed telemetry on LLM calls, embeddings, vector databases, and tool calls. The tool is designed to be local-first, using SQLite with no backend setup required, and it allows users to choose any LLM to interpret the raw evidence and provide plain-English explanations for incidents.\n\nThe project is structured as a pnpm + Turborepo monorepo using TypeScript, with a separate Python package that mirrors the event contract. There are ten event types, including trace, log, metric, error, embedding, vector_op, chain, tool_call, llm_call, and custom. These events carry specific information, such as provider, model, prompt/completion token counts, latency, similarity scores, and query latency. The diagnosis layer examines correlated events for failure patterns, like failed steps, timeouts, zero documents returned, and significant duration regressions. Diagnoses are scored based on evidence volume, statistical deviation, anomalous metrics, deployment correlation, and whether real parent/child span relationships are established. The LLM is used to generate diagnoses from the evidence, with a validation process ensuring the model cites real evidence and not hallucinated information.",
  "summary": "Obyflow is an open-source, CLI-first observability tool for LLM apps, RAG pipelines, agents, vector databases, and LangChain workflows.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}