{
  "id": 18797586,
  "title": "Comment trouver le flux ou les pages de rubrique d'un site d'actualité à partir de robots.txt et des sitemaps en Python",
  "url": "https://urgent.news/2026/11/07/comment-trouver-le-flux-ou-les-pages-de-rubrique-dun-site-dactualite",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-07T21:53:34.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/bestaitools-fr/comment-trouver-le-flux-ou-les-pages-de-rubrique-dun-site-dactualite-a-partir-de-robotstxt-et-4135"
  },
  "original_language": "fr",
  "account": "Finding a news site's feed or category pages involves looking for the feed in the site's resources. Many major news sites don't have a feed, leading some to resort to scraping the homepage, which is fragile and impolite. A better approach exists, and it's hinted at by the site itself: in a link tag, in robots.txt, and in sitemaps. The desktop application osFoundry demonstrates this in Rust (news_feed.rs, functions news_discover_feeds, news_discover_sections, and news_fetch_section). This same process is now available in Python using only the standard library, tested on eight real news sites, showing the number of requests each incurred and where each ended. Basic rules before the first request: a discovery script queries sites it has never seen before. Politeness must be built into the single function that sends requests, not left to the caller. The user agent is Mozilla/5.0 (compatible; osFoundryNews/1.0; +https://osfoundry.io). There's a pause of 1.0 seconds between requests to the same site, with a maximum of 25 requests per site, everything included. An instance of Site is initialized with its host, and it keeps a log, a robots instance, and a last timestamp. The get method fetches a URL with a step parameter. If the log's length is at the MAX_REQUESTS limit, a Budget exception is raised. If the robots.txt file exists for the site and the URL is not robots.txt, the ua checks if the site allows fetching the URL. If it doesn't, a log entry indicating a robots-disallow is added, and None is returned. After waiting the appropriate amount of time, a request is made to the URL with headers specifying the User-Agent and Accept-Encoding. The response status and body are captured, with error handling for HTTP errors and exceptions. If an exception occurs, the status is set to the exception type, and the time since the last request is logged. The process repeats for each site, with additional conditions to stop if robots.txt disallows the site's root for the user agent or if the homepage returns a 401, 403, or 429 status code.",
  "summary": "Le premier travail d'un lecteur de flux, c'est de trouver le flux, et pour beaucoup de grands sites d'actualité il n'y en a pas. La solution de repli à laquelle la plupart des gens pensent, c'est de scraper la page d'accueil, ce qui est fragile et peu poli. Il existe un meilleur ordre pour essayer les choses, et l'essentiel est annoncé par le site lui-même : dans une balise <link> , dans…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}