Turn any web page into clean markdown for ChatGPT or Claude
Paste a web page into ChatGPT or Claude and most of what you send is navigation, cookie banners and script tags. Fetch it with plain requests and a JavaScript-heavy page often comes back nearly empty. Firecrawl solves both: give it a URL and it returns clean markdown of the page's main content, ready for a prompt or a vector store. Here is how to set it up from Python, Node, curl or the CLI, with…
Firecrawl is an API that allows you to convert any web page into clean markdown, HTML, screenshots or structured JSON, making it suitable for use with AI agents like ChatGPT or Claude. The service handles rotating proxies, rate limits, and JavaScript-blocked content, and can also parse PDFs and DOCX files into markdown.
To use Firecrawl, first obtain an API key from firecrawl.dev and store it in an environment variable. Install the Firecrawl Python package, import the required modules, and create a Firecrawl app instance using your API key. Then, call the scrape method on the app object, passing in the URL of the page you wish to convert and specifying the desired output format (markdown in this case). The method will return the clean markdown content of the page.
Firecrawl supports various methods for interacting with web pages, including scraping individual pages, mapping URLs to relevant results, batch scraping multiple pages, and crawling entire websites. This flexibility makes it a powerful tool for creating knowledge bases and providing live web access for coding agents. However, it is essential to respect each site's terms and robots rules when scraping, and be mindful of the costs associated with using the hosted service.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
