Why ChatGPT can't open most web pages (and what to do about it)

When you ask ChatGPT or similar AI about a live web page, you might encounter one of two responses: the model may state it cannot access the page, or it could make a confident guess. This occurs due to several reasons. Firstly, some large websites actively block automated clients like AI tools, resulting in the fetch tools within AI apps receiving a challenge page, an empty site, or a 403 error.

Second, many pages generate their content using JavaScript, leaving little HTML for a simple fetch to retrieve. Thirdly, price and availability information on sites can vary based on the user's location, causing a single fetch from one region to provide inaccurate results. Fourthly, AI chat answers are snapshots, meaning no memory is maintained over time to monitor page changes.

There are limited options for dealing with this issue. One can manually paste the page content, though this is time-consuming and not scalable beyond a single page. Alternatively, one could write a scraper, which works for a single controlled site but often breaks on sites actively blocking bots. Additionally, one could utilize a connector or plug-in that fetches the page with a real browser, returns the facts along with the source link and timestamp, and can schedule re-checks.

Puley is an example of such a tool, which integrates with ChatGPT, Claude, and other AI apps, reads public pages, compares products across countries or stores, turns pages into tables, pulls video transcripts when available, and notifies users when a page changes. However, setup steps per app can be found on puley.com/connect. It should be noted that some sites will still refuse access, and a reliable tool should communicate this rather than guessing.

The effectiveness of an add-on also depends on the AI app, plan, and account. Ultimately, if an AI continues to report it cannot access a page, the problem lies with the fetch step, not the AI model itself. Either you must feed it the content yourself or provide it with a tool capable of opening the page and periodically checking it.