Apple's App Store review feed has holes — and whether you hit one depends on your HTTP client

Apple's App Store review feed has inconsistencies and accessibility depends on the HTTP client being used, according to recent findings. The feed, which can be accessed via the URL itunes.apple.com/country/rss/customerreviews/id=.../sortBy=.../page=n/json, appears to have empty pages appearing randomly throughout a result set, rather than at the end. These empty pages can cause scrapers to prematurely end their data collection, reporting "no reviews found" when, in fact, reviews exist.

The issue is not limited to a single device or browser, but rather the HTTP client used to make the request. For example, a curl request made with a plain HTTP client may return empty pages consistently, while a browser-shaped client with browser-like headers can retrieve the complete set of reviews. This discrepancy between curl and browser headers suggests that the client's shape, rather than the Apple device itself, determines whether a specific page is a hole in the feed.

The original advice to rotate User-Agent strings to handle empty pages and cache the winning variant proved ineffective. Not only did this approach result in up to four times the number of requests on every empty page, it also concealed the actual cause of the problem. When a scraper encounters four fingerprints and still sees no reviews, it may falsely conclude that the feed is truly empty. In reality, the issue lies in the loop prematurely stopping at page one and never checking subsequent pages.

To accurately access the App Store review feed, it is recommended to test the anti-bot theory before implementing any scraping machinery. Additionally, it is crucial to test the solution on different days, as the feed's state can change over time. Varying the client class, such as using curl versus real-browser headers, can help predict the presence or absence of holes in the feed.