What I Learned Turning an AI News Crawler into a Working Product
Fetching an article is only the beginning of a news pipeline. The harder work is deciding whether the response is an article, whether it belongs in the product, and whether a reader can understand what was published. I am building yocho, an AI industry news and analysis service. This is an architecture note about the work between acquisition and publication, based on the implementation and…
Once an article is fetched, the real work begins in determining if it belongs in the product and if a reader can comprehend it. Yocho, an AI news and analysis service, aims to combine category-based news browsing with Japanese editorial articles and public analysis posts within one application. This architecture note focuses on the pipeline between acquisition and publication, based on implementation and operational checks from September 9, 2026.
The backend employs FastAPI and Python for database access and domain logic, while PostgreSQL stores application data and job queues, and GCS holds raw archives. The frontend interacts with the backend rather than connecting directly to the database, ensuring consistent publication rules across various news pages and background processing.
Fetching is a critical milestone, as a successful response merely confirms the server returned something, not that the response contains a usable article. The pipeline consists of several distinct decisions, each capable of failure and requiring its own observable outcome. For instance, a URL might return a listing page with many headlines, which, if treated as a single article, would create an incorrect record with the wrong content unit. Another page might contain a real article but be unsuitable for the service's editorial scope.
Classification poses challenges, such as distinguishing between energy storage and computer memory despite the shared word "storage." A useful regression case demands context to make meaningful category distinctions. Testing confusing neighbors is crucial, as a classifier may appear reasonable on obvious positive examples while still damaging the reader's experience at category boundaries. Developers should collect a small set of near misses before expanding their source list.
Retries are equally important. The current automation generates both the article body and a dedicated feed summary, committing publication and task completion within the same database transaction. This approach prevents independent recording of database outcomes, meaning operational checks cannot reliably detect rate limiting or subsequent successful processing.
Recorded rate limiting and recovery support a bounded statement about workflow resilience but do not guarantee the entire backlog's completion or automatic handling of future provider failures. Developers should evaluate whether a subsequent worker can distinguish work needing retry from work already published before increasing concurrency.
The public product now includes more of the reading journey, but completeness and quality remain separate goals. While more source coverage does not guarantee a useful stream in every category, a generated summary must also undergo review to avoid repetition. An important test involves preserving sources and interpretation separately, allowing readers to recover original evidence and explain what would change their conclusions.
Yocho provides a starting point for testing this approach by inspecting a news item's title, category, editorial text, and source link as separate outputs.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
