{
  "id": 14224030,
  "title": "My URL-to-Markdown converter turned a paywalled article into free text — and I had to decide if that was a bug",
  "url": "https://urgent.news/2026/10/13/my-url-to-markdown-converter-turned-a-paywalled-article-into-free",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-13T15:02:51.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/imapphelp/my-url-to-markdown-converter-turned-a-paywalled-article-into-free-text-and-i-had-to-decide-if-18ai"
  },
  "original_language": "en",
  "account": "In the course of developing a URL-to-Markdown API, the author encountered an unexpected situation. While testing their system, they fed it a link to a paywalled news article. To their surprise, the API successfully returned the entire article, including the content that was intended to be hidden behind a client-side overlay and paywall.\n\nThe conversion process worked flawlessly, stripping away the usual navigation, advertisements, and sidebars, and delivering the main article body as clean, LLM-ready Markdown. However, the author found themselves pondering over the ethical implications of this discovery.\n\nThe paywall's existence is determined by server-side factors, such as CSS and JavaScript, rather than server-side authentication checks. This meant that the converter, when designed to extract whatever the server sends, would inadvertently retrieve the full text of paywalled articles. The author grappled with the decision of whether to implement a fix for this issue, or to maintain their original approach of simply returning what the server provided.\n\nUltimately, the author opted for a middle ground. They introduced a toggle in their converter, allowing users to switch between a raw mode and a readability mode. In raw mode, the converter would return whatever the server sends, including potentially paywalled content. Readability mode, on the other hand, would strip away navigational elements and return a clean, LLM-ready version of the article. The author also included detailed documentation explaining that the readability mode reflects the server's decision to include or exclude content, and that it doesn't constitute a breach of access controls.\n\nThis experience reinforced the author's understanding that clean extraction and respecting access controls are two separate features. They had only focused on the former, assuming it automatically encompassed the latter. The lesson learned was clear: publishers need to take responsibility for enforcing paywalls on their end, rather than leaving it up to the extractor. The author made their tool available online, providing a selector parameter to target specific regions if desired.",
  "summary": "I built a URL-to-Markdown API that fetches any web page and returns the main content as clean, LLM-ready Markdown. The readability extraction worked exactly as designed: strip nav, ads, sidebars, keep the article body. Test suite green. Then my human pasted a link to a paywalled news article and the API returned the full story text. Here's what happened. The site serves the article body in the…",
  "key_points": [
    "Paywalled article fully converted into clean Markdown",
    "Server-side factors determine paywall existence, not authentication",
    "Author created toggle for raw vs readability modes in converter"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}