{
  "id": 10175299,
  "title": "Three things that cost me real time building a 25-source news aggregator",
  "url": "https://urgent.news/2026/09/27/three-things-that-cost-me-real-time-building-a-25-source-news",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-27T08:16:16.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/beaconxuli/three-things-that-cost-me-real-time-building-a-25-source-news-aggregator-27mg"
  },
  "original_language": "en",
  "account": "Building a 25-source news aggregator consumed far more effort than expected. The aggregation process itself proved relatively straightforward. However, three aspects consumed the real time:\n\n1. Google News RSS links present JavaScript shells. Directly fetching the URLs yields only placeholder pages that resolve the actual publisher URLs client-side. To obtain the real article summary, one must:\n- Fetch the shell page,\n- Extract two attributes (data-n-a-sg and data-n-a-ts),\n- POST them along with the article ID to Google's internal batchexecute endpoint,\n- Scrape the real URL from the response.\n\nThe request body involves a nested array serialized as f.req. Implementing this solution is undocumented and prone to breaking without notice. The process should be guarded, returning an empty string on failure instead of guessing a URL. Even when the link is resolved, expect partial success as some publishers block subsequent fetches. In the author's case, about a third of Google News items lacked an in-site summary due to publisher access restrictions, not the resolver's fault.\n\n2. Static hosting rebuilds quietly consume the monthly budget. The initial design was correct: a static site updates when new items appear in the store. However, the numbers revealed the true cost. Netlify charges 15 credits per production deploy, with a plan that includes 1000 credits per billing period. A daily rebuild translates to 15 credits × 30 days = 450 credits, nearly half the budget allocated for publishing headlines. The collector, running more than once a day practically, further exacerbated the expense. The solution was to differentiate \"fetch new data\" from \"rebuild the site.\" The collector writes to the store without triggering a build. Pages ship with a static snapshot for crawlers and first paint efficiency. Upon loading, the page checks if the store is newer than the snapshot; if so, it re-renders the list client-side using the API endpoint `/api/news/list?days=10&per=60`. If the API is unreachable, the site gracefully falls back to the static snapshot. This adjustment reduced daily updates to zero deploys, spending credits only when code changes occur. The trade-off is evident: the static snapshot ensures SEO and first paint authority, while client-side updates provide progressive enhancement.",
  "summary": "I built a small forestry news aggregator: 25 sources, rolling 10-day window, every headline machine-translated into Chinese/English/Spanish, every item linking back to the publisher. It runs on Astro (static) + Netlify Functions + Netlify Blobs as a key-value store. The aggregation itself was the easy part. These three things were not. 1. Google News RSS links are JavaScript shells If you've ever…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}