{
  "id": 16631200,
  "title": "Your crawler sees an empty div. Crawling a JavaScript SPA with Playwright",
  "url": "https://urgent.news/2026/10/26/your-crawler-sees-an-empty-div-crawling-a-javascript-spa-with",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-26T17:01:12.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/branch42-labs/your-crawler-sees-an-empty-div-crawling-a-javascript-spa-with-playwright-2aed"
  },
  "original_language": "en",
  "account": "The source material describes how to crawl a JavaScript Single Page Application (SPA) using Playwright. The HTML sent by the server for such a site is minimal, containing just a script tag and an empty div with id \"root\". Crawlers that parse this HTML find no links to follow, causing the crawl to stop. To overcome this, each page must be loaded in a headless browser, which renders the application and extracts hyperlinks from the live DOM. Playwright and Puppeteer are examples of such browsers.\n\nThe crawler should also respect a website's robots.txt file, which specifies which pages can be accessed. The maximum number of pages to crawl, page load delay, and whether to ignore certain routes can be configured. After setting up the Playwright environment, the crawler begins by fetching the site's robots.txt file and parsing it for rules. It then initializes a browser instance and a new page, setting up an event listener for resource requests.\n\nAs the crawler visits each URL, it loads the page, waits for the DOM to load, and extracts hyperlinks. These hyperlinks are stored in a queue, with duplicates removed. The crawler continues this process until it has visited the maximum number of pages or exhausts the queue. It also checks if the crawler is allowed to access a page based on the robots.txt rules. If a page fails to load, the crawler logs the error and continues.",
  "summary": "TL;DR: On a client-rendered SPA, the HTML the server sends is a script tag and an empty <div id=\"root\"> . An HTTP crawler parses that, finds no links and stops. Load each page in a headless browser (Playwright or Puppeteer), wait for it to render, read a[href] from the live DOM and queue what you find. If you want the data rather than the pages, the JSON API the app calls is usually the better…",
  "key_points": [
    "JavaScript SPA contains minimal HTML with script tag and empty div",
    "Playwright and Puppeteer render SPA to extract hyperlinks",
    "Crawler respects robots.txt, configures page limits and delays"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}