Your crawler sees an empty div. Crawling a JavaScript SPA with Playwright
TL;DR: On a client-rendered SPA, the HTML the server sends is a script tag and an empty <div id="root"> . An HTTP crawler parses that, finds no links and stops. Load each page in a headless browser (Playwright or Puppeteer), wait for it to render, read a[href] from the live DOM and queue what you find. If you want the data rather than the pages, the JSON API the app calls is usually the better…
The source material describes how to crawl a JavaScript Single Page Application (SPA) using Playwright. The HTML sent by the server for such a site is minimal, containing just a script tag and an empty div with id "root". Crawlers that parse this HTML find no links to follow, causing the crawl to stop. To overcome this, each page must be loaded in a headless browser, which renders the application and extracts hyperlinks from the live DOM. Playwright and Puppeteer are examples of such browsers.
The crawler should also respect a website's robots.txt file, which specifies which pages can be accessed. The maximum number of pages to crawl, page load delay, and whether to ignore certain routes can be configured. After setting up the Playwright environment, the crawler begins by fetching the site's robots.txt file and parsing it for rules. It then initializes a browser instance and a new page, setting up an event listener for resource requests.
As the crawler visits each URL, it loads the page, waits for the DOM to load, and extracts hyperlinks. These hyperlinks are stored in a queue, with duplicates removed. The crawler continues this process until it has visited the maximum number of pages or exhausts the queue. It also checks if the crawler is allowed to access a page based on the robots.txt rules. If a page fails to load, the crawler logs the error and continues.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
