{
  "id": 7290920,
  "title": "Scraping Used to Take a Programmer. Now It Takes a Sentence.",
  "url": "https://urgent.news/2026/09/14/scraping-used-to-take-a-programmer-now-it-takes-a-sentence",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-14T02:52:58.000Z",
  "source": {
    "name": "HackerNoon",
    "slug": "hackernoon",
    "url": "https://hackernoon.com/scraping-used-to-take-a-programmer-now-it-takes-a-sentence?source=rss"
  },
  "original_language": "en",
  "account": "In recent years, the methods used to scrape data from websites have evolved significantly. Scrapers, which previously required programmers to manually extract data by targeting specific paths within a page's HTML structure, are now being taken over by large language models. This shift is due to the fact that AI models are exceptionally skilled at handling the technical challenges that once deterred amateurs from scraping data at scale.\n\nA scraper typically performs four main steps: requesting a page, receiving the response, extracting the desired information, and moving on to the next page. However, this process has become increasingly complex due to three complications. Firstly, many modern websites do not display the data they contain in the initial HTML chunk but rather fetch it from an internal API using JavaScript. Secondly, navigating through a website with features like \"next\" buttons, infinite scroll, and links that lead to more pages requires expertise in web development. Lastly, selectors used to extract data are fragile and break when a site renames a class or reshuffles its layout, necessitating constant maintenance.\n\nArtificial intelligence has significantly mitigated these challenges. Language models, trained on vast amounts of code and HTML, can generate the necessary extraction logic when given a description of the desired data in plain language. This allows users to bypass the need for understanding HTML structure and write code. Additionally, AI can understand the meaning of data rather than relying on specific HTML tags, making it resilient to changes in the website's layout. Furthermore, AI-driven scrapers can autonomously navigate websites, interpreting their content and adapting to changes, thus preserving the extracted data without requiring constant maintenance.\n\nWith AI handling the technical aspects of scraping, the cost of entry has plummeted. Starting to scrape data is now as simple as using a chat box or making a cheap API call. The barrier to entry has been lowered to nearly nothing, and the cost per record collected is a fraction of a cent. Consequently, high-volume data collection is no longer limited to a small group of skilled programmers, but is now accessible to a broader range of users.",
  "summary": "AI didn't invent web scraping. It removed the one thing that kept it hard: needing to know how. Here's which defenses still hold, and which ones just broke.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}