{
  "id": 7621371,
  "title": "I Built a Page Importer That Clones Any URL Into an Editor",
  "url": "https://urgent.news/2026/09/15/i-built-a-page-importer-that-clones-any-url-into-an-editor",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-15T20:01:02.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/knot_crochet_dbb4379fde5d/i-built-a-page-importer-that-clones-any-url-into-an-editor-go1"
  },
  "original_language": "en",
  "account": "The Page Importer is a tool that creates a duplicate of any public webpage in an editor, complete with all images, stylesheets, and scripts. It is initiated by a button within the editor, and the process takes only a few seconds. The implementation involves several steps, each serving a specific purpose. First, there's a fetch operation to retrieve the HTML from the server. However, this method doesn't capture any client-side rendering, which is common in modern marketing pages. Therefore, a second step is employed, using a headless browser API to generate the DOM after the page has loaded. The most crucial aspect is the fallback policy, as it determines how the system handles errors during the import process. Three different strategies are employed based on the type of error encountered. Firstly, if the browser rendering is not configured, the system proceeds with the plain fetch method and logs the event at the info level. Secondly, if there's a network failure, the system degrades gracefully and logs the error at the warn level. Lastly, if an HTTP error originates from the browser service, it propagates as a 502 error. The result of the import process is labeled with the tier that produced it, enabling the UI to display which path was taken. The system also includes a bot wall detector to prevent the creation of a page containing someone else's captcha. This detector utilizes string checks on specific patterns like \"title Just a moment...\" or \"class= cf-browser-verification\". Additionally, there's a size-gated rule to avoid false positives from real pages that might contain the detection keywords. The assets within the captured page often number between 50 and 150, necessitating a method to handle their download, upload to the user's storage, and rewriting of the HTML references without exceeding the request timeout. The system achieves this by collecting all absolute source URLs first, uploading unique ones using a bounded pool with a concurrency limit of 5, and then rewriting the HTML synchronously from the resolved map. This approach ensures a comprehensible and efficient process, minimizing the chances of encountering network-related issues.",
  "summary": "\"Import from URL\" is a button in my page editor. You paste any public page's address, and a few seconds later an editable copy of it is open in the canvas, with every image, stylesheet and script already served from your own CDN. The feature description is one sentence. The implementation is a fetch, a bot-wall detector, an HTML parser, a bounded upload pool, and a network egress policy. Here's…",
  "key_points": [
    "Page Importer duplicates public webpages in an editor",
    "Uses fetch and headless browser for full page capture",
    "Fallback policies handle errors with info, warn, and 502 logs"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}