{
  "id": 1478857,
  "title": "I read the Terms of Service for every source in my news aggregator",
  "url": "https://urgent.news/2026/08/17/i-read-the-terms-of-service-for-every-source-in-my-news-aggregator",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-17T12:01:31.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/madareyou/i-read-the-terms-of-service-for-every-source-in-my-news-aggregator-14j0"
  },
  "original_language": "en",
  "account": "I constructed a daily tech digest called devdigest. It compiles content from RSS feeds, employs a model to categorize and rate the excerpts, generates original summaries, and then sends subscribers a list of titles, summaries, and hyperlinks to the original articles. The full-text of any article is never reproduced or stored beyond a temporary cache during the run. To read the articles, you must click through to the publisher's site. Before monetizing the service, I wanted to ensure that none of my sources explicitly prohibited automated access. Consequently, I read the actual Terms of Service (ToS) for all of them. I was surprised to find that my hypothesis about the size of the companies was incorrect. In fact, the most restrictive terms were from VC-data companies and large corporate infrastructure blogs, while the permissive terms came from smaller publications and well-lawyered commercial sites. The findings revealed three potential pitfalls when determining the availability of content:\n\n1. The Terms of Service can be clean, but the robots.txt file may still prohibit access. For instance, Towards Data Science had a clean ToS but explicitly banned AI crawlers like ClaudeBot and GPTBot in its robots.txt file. Ignoring robots.txt led me to remove a valuable source.\n\n2. A site may publish an RSS feed, but its ToS still prohibits scraping, even if the feed itself is allowed. Sifted, for example, provided an RSS feed link on its homepage but explicitly prohibited text and data mining, including the use of automated devices like robots, bots, spiders, and scrapers. Simply reading the existence of the feed as permission was incorrect; the Terms of Use were the definitive document.\n\n3. The scraping clause may be scoped differently than expected. TechCrunch, for instance, had a robots/spiders/scrapers clause that seemed to indicate a ban on publishing articles. However, upon closer examination, the clause only applied to user-submitted content, comments, and forums, while the dedicated RSS terms explicitly permitted the use of published articles. Failing to read the full context of the clause led to the exclusion of a source that actually supported my use case.\n\nThe key takeaway is that commercial media ToS often contain a generic statement like \"all rights reserved, no reproduction without permission,\" which appears to be a universal prohibition. However, this boilerplate language does not actually cover the specific activities of RSS readers, feed aggregators, and newsletters. To accurately assess the feasibility of using a source, look for clauses that explicitly mention automated access, bots, scraping, RSS or feed reuse, or AI training and summarization. These terms directly relate to the pipeline used in the digest creation process. Generic anti-plagiarism language should not be relied upon as a definitive answer.",
  "summary": "I build devdigest , a daily tech digest. It pulls from RSS feeds, has a model read each excerpt to categorise and score it, writes an original summary, and emails subscribers a list of titles, summaries and links back to the original articles. Full article text is never reproduced or stored beyond a short-lived per-run cache. To read anything, you click through to the publisher. Before charging…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}