4.5B Posts Scraped from TikTok

In a three-week period, a system utilizing TikTok's private HTTP+JSON API gathered an extensive dataset, collecting 3.23 billion creator profiles, 5.94 billion videos, and 2.8 billion comments. This comprehensive dataset, containing captions, view, like, comment, and save counts, sound, country, and posting time, has been made publicly available for further exploration and analysis.

The dataset covers not only video details but also creator profiles, follower and following lists, similar creators graph, full video statistics, comments and replies with account information, sounds and trending sounds, hashtags along with associated videos, keyword searches across videos, creators, and sounds, trending shelves, and camera effects. The dataset can be accessed directly from Hugging Face at huggingface.co/datasets/kuben-developer/tiktok-videos-4b.

To access TikTok's private API, one must first register a device using a unique device credential issued by TikTok. Subsequent requests require a valid request signature, the correct regional host, and a TLS handshake resembling that of a mobile device. Failure to meet any of these four criteria results in a successful HTTP 200 response with an empty body, making it difficult to identify the specific failure point.

The API provides access to various endpoints, each with a measured success rate. Some endpoints, such as fetching a full profile record, may return an empty response even when successful. This empty response indicates a soft block, meaning that access to certain account-gated features, such as direct messages, private videos, or who liked what, is not possible.