Skip to main content

Extract every URL from a sitemap.xml in Python (indexes, .gz, robots.txt) and spot competitors' new pages

A site's sitemap is the cheapest full list of its pages you will ever get: the site publishes it for search engines, it usually carries a lastmod date per URL, and reading it takes seconds instead of crawling HTML for hours. That makes it the right starting point for SEO audits, migrations, content inventories, feeding a crawler or RAG pipeline, and keeping an eye on what competitors publish. The…

dev.to ·4h ago

We haven't written up this one. Dev.to has the full story — the link below goes straight to it.

Read the original at dev.to →

More in Tech

How AI cuts 50% of bitrate in 8K real-time streaming

📈 Tendência explosiva: IA reduz em até 50 % o bitrate do streaming 8K em tempo real Introdução Nos últimos meses, as buscas por “8K streaming” e “compressão de vídeo com IA” dispararam no Google…

AI reduces 8K streaming bitrate by 50% without quality loss
Models like Transformers and Diffusion learn to predict future frames and eliminate redundancies
Implementation allows transmitting 8K videos at 40 Mbps, half of traditional codecs

4h ago ·dev.to

Roblox Image Sizes in 2026: Thumbnails, Icons, Badges and Game Passes (Cheat Sheet)

If you've ever uploaded a Roblox thumbnail and watched your title get cut off, or uploaded a game pass icon and realised the important part got cropped into a circle, this one's for you.

4h ago ·dev.to

Ridley Scott's 2026 Post-Apocalyptic Thriller Officially Sets Home Video Release

Ridley Scott’s 'The Dog Stars' is coming to digital streaming and purchase VOD platforms on Oct. 27, with a physical release in December.

4h ago ·polygon.com

AMD is bringing FSR 4.1 to Xbox consoles and handhelds

AMD has just revealed that it's bringing its latest upscaling technology, FSR 4.1, to the Xbox Series S and X consoles, as well as laptops, 6000-series GPUs, and handhelds like the Xbox Ally X…

4h ago ·theverge.com

How Much Should a US Freelancer Set Aside for Taxes? (Self-Employment Tax + Quarterly Payments Explained)

If you just went from a W-2 job to freelancing (or picked up 1099 side gigs), the first client payment feels great. Then you realize nobody withheld anything.

4h ago ·dev.to

The drift a scanner can't see: when the instruction is stale, not hostile

Dhanush N's agentcheck targets a real thing: instructions that arrive carrying an attack. A tag-character payload, a poisoned rule, a server that rewrites its own tool list.

Stale instructions describe original agent state, not current one
Staleness stems from behavior, not text content
Detecting drift requires behavior-based checks, costly to implement

4h ago ·dev.to

More from Tuesday 20 October →