Three things that cost me real time building a 25-source news aggregator
I built a small forestry news aggregator: 25 sources, rolling 10-day window, every headline machine-translated into Chinese/English/Spanish, every item linking back to the publisher. It runs on Astro (static) + Netlify Functions + Netlify Blobs as a key-value store. The aggregation itself was the easy part. These three things were not. 1. Google News RSS links are JavaScript shells If you've ever…
Building a 25-source news aggregator consumed far more effort than expected. The aggregation process itself proved relatively straightforward. However, three aspects consumed the real time:
1. Google News RSS links present JavaScript shells. Directly fetching the URLs yields only placeholder pages that resolve the actual publisher URLs client-side. To obtain the real article summary, one must:
- Fetch the shell page,
- Extract two attributes (data-n-a-sg and data-n-a-ts),
- POST them along with the article ID to Google's internal batchexecute endpoint,
- Scrape the real URL from the response.
The request body involves a nested array serialized as f.req. Implementing this solution is undocumented and prone to breaking without notice. The process should be guarded, returning an empty string on failure instead of guessing a URL. Even when the link is resolved, expect partial success as some publishers block subsequent fetches. In the author's case, about a third of Google News items lacked an in-site summary due to publisher access restrictions, not the resolver's fault.
2. Static hosting rebuilds quietly consume the monthly budget. The initial design was correct: a static site updates when new items appear in the store. However, the numbers revealed the true cost. Netlify charges 15 credits per production deploy, with a plan that includes 1000 credits per billing period. A daily rebuild translates to 15 credits × 30 days = 450 credits, nearly half the budget allocated for publishing headlines.
The collector, running more than once a day practically, further exacerbated the expense. The solution was to differentiate "fetch new data" from "rebuild the site." The collector writes to the store without triggering a build. Pages ship with a static snapshot for crawlers and first paint efficiency. Upon loading, the page checks if the store is newer than the snapshot; if so, it re-renders the list client-side using the API endpoint `/api/news/list?days=10&per=60`.
If the API is unreachable, the site gracefully falls back to the static snapshot. This adjustment reduced daily updates to zero deploys, spending credits only when code changes occur. The trade-off is evident: the static snapshot ensures SEO and first paint authority, while client-side updates provide progressive enhancement.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
