What it actually takes to read a Xiaohongshu post from a server
Originally published at linkdigest.dev , where I build this. The problem, in one paragraph Paste a Xiaohongshu, Douyin, TikTok, YouTube or X link into an AI agent and it fetches the URL, gets an app-download shell or a login wall, and tells you there is nothing there. It is not wrong. The content of those posts is video, images, and text printed inside images, behind tokenised share links — none…
Xiaohongshu posts require specific extraction methods to be read by AI agents. When a post contains videos, images, and text printed inside images, the HTML fetch returns none of this content. To overcome this, LinkDigest provides a hosted reader that turns the link into text with timecodes, on-screen text, image OCR, and metadata. The service uses an MCP server, REST API, and web console to process and deliver the information in Markdown or JSON format.
The service handles various platforms, including Bilibili, Instagram, Facebook, and YouTube, but each has its limitations. For Xiaohongshu, the extractor parses the state blob and imageList from the page payload, which differs depending on the user agent requested. Anonymous requests fail, but fetching the /explore feed first and keeping the cookie jar allows access.
Douyin demands cookies from a logged-in session, while YouTube blocks datacenter IP ranges, necessitating a residential proxy. The service also addresses failure scenarios, ensuring that a thin result is distinguishable from an empty result and implementing a short TTL for thin results to limit cache staleness and costs.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
