My URL-to-Markdown converter turned a paywalled article into free text — and I had to decide if that was a bug

In the course of developing a URL-to-Markdown API, the author encountered an unexpected situation. While testing their system, they fed it a link to a paywalled news article. To their surprise, the API successfully returned the entire article, including the content that was intended to be hidden behind a client-side overlay and paywall.

The conversion process worked flawlessly, stripping away the usual navigation, advertisements, and sidebars, and delivering the main article body as clean, LLM-ready Markdown. However, the author found themselves pondering over the ethical implications of this discovery.

The paywall's existence is determined by server-side factors, such as CSS and JavaScript, rather than server-side authentication checks. This meant that the converter, when designed to extract whatever the server sends, would inadvertently retrieve the full text of paywalled articles. The author grappled with the decision of whether to implement a fix for this issue, or to maintain their original approach of simply returning what the server provided.

Ultimately, the author opted for a middle ground. They introduced a toggle in their converter, allowing users to switch between a raw mode and a readability mode. In raw mode, the converter would return whatever the server sends, including potentially paywalled content. Readability mode, on the other hand, would strip away navigational elements and return a clean, LLM-ready version of the article.

The author also included detailed documentation explaining that the readability mode reflects the server's decision to include or exclude content, and that it doesn't constitute a breach of access controls.

This experience reinforced the author's understanding that clean extraction and respecting access controls are two separate features. They had only focused on the former, assuming it automatically encompassed the latter. The lesson learned was clear: publishers need to take responsibility for enforcing paywalls on their end, rather than leaving it up to the extractor. The author made their tool available online, providing a selector parameter to target specific regions if desired.