Urgent.News

What's breaking now, across thousands of outlets.

Tech

My Snapchat scraper returned 0 results for a day and still said "SUCCEEDED"

On Monday I checked my Snapchat Spotlight scraper and it looked fine. Green status, run finished in five seconds, exit code 0. The dataset was empty. It had been empty since Snapchat pushed a change to their website, and nothing told me. No error, no failed run. If I hadn't opened the output by hand I would have kept "selling" a tool that returned nothing. Here's what broke, how I found the data…

On Monday, the user checked their Snapchat Spotlight scraper and everything appeared to be working fine. The status was green, the run finished quickly, and the exit code was 0. However, upon opening the output manually, they discovered that the dataset was empty. This was because Snapchat had changed their website layout, but the user was unaware of it. They only noticed the issue after manually inspecting the output.

The user then investigated what had changed. Snapchat's public profiles (snapchat.com/@username) used to contain data within a script tag. However, this changed when the profile pages moved to the Next.js App Router. The script tag (__NEXT_DATA__) was no longer present. When the scraper encountered this change, it raised an error, caught it, logged it, and moved on to the next URL.

In this case, with only one URL to process, moving on meant finishing with zero items and a successful exit code. This demonstrated the importance of checking the actual data rather than just the run status.

The user found that the data had moved to the HTML of the page, where it was streamed as React Server Components. They discovered that the video data was in the HTML as normal JSON objects, which could be extracted by searching for a marker and decoding one JSON value at a time using Python's json.JSONDecoder().raw_decode. This method allowed them to ignore any additional content that followed the JSON value.

Another bug was discovered when the user noticed that some Spotlight videos had empty caption files, but the audio was still present and transcribed by Whisper without issues. The user realized that an empty caption file did not necessarily mean silence and suggested treating both empty and missing caption files the same way, allowing Whisper to make the final decision on whether there was actually speech or not.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Why a Working App Can Still Have the Wrong Database Schema

AI-generated database schemas—the structures that define what information a database stores and how that information relates—can look convincing.

  • Database schema not always necessary
  • AI-generated stats table suggested
  • Data collection may be unnecessary

Terminal recordings, reimagined

"I'll remember!" Famous last words. You spend two hours chasing a bug, hardening a server or deploying a service. The veteran in you knows you should document the steps as you go.

  • Trecs records terminal activity, preserving commands, outputs, and visuals
  • User-friendly interface allows easy navigation and search within recordings
  • Open-source (AGPLv3) and statically linked, available for various platforms

A Token Swap Is a Distributed Systems Problem Wearing a Simple UI

TronSeven — Non-Custodial Token Swaps on TRON Network Swap TRC-20 tokens on TRON with self-custody and open-source smart contracts. Low fees, verifiable on TRONSCAN.

  • Token swap is complex distributed systems problem with financial stakes.
  • Users must connect wallet, identify token contract, authorize operation.
  • Model swap as state machine with preparation, quotation, authorization stages.

More from Tuesday 17 November →