How We Scanned 1 Million Solana Token Accounts Without Timing Out

Our Holder Snapshot tool identifies token holders and their balances. Projects use it for airdrops, governance voting, and verifying token holder counts. For most tokens, fetching token accounts, filtering out empty ones, grouping by owner, and summing balances is straightforward. However, BONK's token had one million accounts, causing our initial implementation to time out.

The scan took longer per page as we progressed, and eventually, the API refused requests and recommended cursor-based pagination. Our original approach used sequential page fetching, which proved inefficient due to increasing retrieval times and potential timeouts. We also had to contend with rate limits, requests hanging indefinitely, token balances exceeding JavaScript's safe integer range, and CPU-intensive operations freezing the browser tab.

To address these issues, we redesigned the scanner. First, counting token holders proved more complex than it appeared. Unlike wallets holding balances directly, token balances reside in token accounts associated with a mint and owner. Empty accounts may persist until closed, necessitating additional steps to calculate the actual holder count.

We fetched all token accounts for a mint, discarded zero-balances, grouped the remaining accounts by owner, and summed balances per owner. However, retrieving every account efficiently posed a challenge. Our initial pagination method using getProgramAccounts with a memcmp filter on the mint worked for smaller datasets but struggled with tokens having a million accounts.

Many RPC providers limit or disable this method for large mints. Switching to Helius' DAS API and its getTokenAccounts endpoint provided a paginated alternative, returning up to 1,000 accounts per request while allowing exclusion of zero-balances. We initially requested page 1, then 2, then 3, continuing until exhausted. However, this approach failed for BONK, as requests around page 200 timed out after 15 seconds due to an upstream database timeout.

After passing 500,000 accounts, the API recommended cursor-based pagination. The issue lay in offset pagination, where retrieving later pages became increasingly expensive as the database had to skip more records. To move through the dataset more efficiently, we adopted keyset pagination using Helius' getTokenAccounts cursor parameter along with after and before bounds on account addresses.

This approach eliminated the growing cost of offset pagination by requesting the next 1,000 accounts after a specific address instead of page numbers. Still, a cursor alone wasn't enough, as each request depended on the previous one. To parallelize the scan and avoid cumulative delays, we estimated BONK's token size before scanning everything.

Using position and edge functions based on Solana address bytes, we divided the address space into independently scannable ranges. This enabled parallel processing and significantly improved the efficiency of scanning 1 million Solana token accounts without timeouts.