{
  "id": 6381507,
  "title": "What I Learned Turning an AI News Crawler into a Working Product",
  "url": "https://urgent.news/2026/09/09/what-i-learned-turning-an-ai-news-crawler-into-a-working-product",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-09T04:38:21.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/hikaru_sato_ce8ce2cff7c01/what-i-learned-turning-an-ai-news-crawler-into-a-working-product-i6g"
  },
  "original_language": "en",
  "account": "Once an article is fetched, the real work begins in determining if it belongs in the product and if a reader can comprehend it. Yocho, an AI news and analysis service, aims to combine category-based news browsing with Japanese editorial articles and public analysis posts within one application. This architecture note focuses on the pipeline between acquisition and publication, based on implementation and operational checks from September 9, 2026. The backend employs FastAPI and Python for database access and domain logic, while PostgreSQL stores application data and job queues, and GCS holds raw archives. The frontend interacts with the backend rather than connecting directly to the database, ensuring consistent publication rules across various news pages and background processing.\n\nFetching is a critical milestone, as a successful response merely confirms the server returned something, not that the response contains a usable article. The pipeline consists of several distinct decisions, each capable of failure and requiring its own observable outcome. For instance, a URL might return a listing page with many headlines, which, if treated as a single article, would create an incorrect record with the wrong content unit. Another page might contain a real article but be unsuitable for the service's editorial scope.\n\nClassification poses challenges, such as distinguishing between energy storage and computer memory despite the shared word \"storage.\" A useful regression case demands context to make meaningful category distinctions. Testing confusing neighbors is crucial, as a classifier may appear reasonable on obvious positive examples while still damaging the reader's experience at category boundaries. Developers should collect a small set of near misses before expanding their source list.\n\nRetries are equally important. The current automation generates both the article body and a dedicated feed summary, committing publication and task completion within the same database transaction. This approach prevents independent recording of database outcomes, meaning operational checks cannot reliably detect rate limiting or subsequent successful processing. Recorded rate limiting and recovery support a bounded statement about workflow resilience but do not guarantee the entire backlog's completion or automatic handling of future provider failures. Developers should evaluate whether a subsequent worker can distinguish work needing retry from work already published before increasing concurrency.\n\nThe public product now includes more of the reading journey, but completeness and quality remain separate goals. While more source coverage does not guarantee a useful stream in every category, a generated summary must also undergo review to avoid repetition. An important test involves preserving sources and interpretation separately, allowing readers to recover original evidence and explain what would change their conclusions. Yocho provides a starting point for testing this approach by inspecting a news item's title, category, editorial text, and source link as separate outputs.",
  "summary": "Fetching an article is only the beginning of a news pipeline. The harder work is deciding whether the response is an article, whether it belongs in the product, and whether a reader can understand what was published. I am building yocho, an AI industry news and analysis service. This is an architecture note about the work between acquisition and publication, based on the implementation and…",
  "key_points": [
    "Yocho AI service combines news browsing with Japanese editorial articles",
    "Backend uses FastAPI, Python, PostgreSQL, and GCS for data storage and processing",
    "Classification challenges arise from distinguishing similar topics like energy storage"
  ],
  "editors_take": "The development highlights the complexity of processing AI-fetched articles for publication, underscoring the need for multiple checks and balances to ensure quality and relevance in a news aggregation service.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}