API Pagination Patterns: Offset, Cursor, and Keyset Explained

Pagination is a necessary feature for APIs that return lists of data. When a dataset becomes large, naive approaches like passing page numbers and a limit quickly lead to performance issues and consistency problems. This article explains the three main pagination patterns - offset, cursor, and keyset - their pros and cons, and when to use each.

Offset pagination is the simplest approach. It uses an 'offset' parameter indicating how many records to skip, and a 'limit' parameter specifying how many records to return. Implementing offset pagination is straightforward and intuitive for developers. A sample implementation using Node.js and a database query is provided in the source.

The main advantages of offset pagination are ease of implementation and straightforward navigation to any page. However, it has significant drawbacks for large datasets. Deep offsets cause the database to scan and discard rows before reaching the desired offset, resulting in slow performance. Additionally, with concurrent writes, offset pagination can return duplicate records or miss records altogether, leading to inconsistencies. It's also not suitable for real-time feeds or infinite scroll scenarios.

Cursor pagination represents the modern standard for paginating API responses. Instead of counting rows, a cursor is passed, which is an opaque value representing the position of the last item viewed. The server decodes this cursor and queries the database based on the decoded cursor value. A Node.js example of cursor pagination is shown in the source.

Cursor pagination offers several benefits. It remains fast regardless of the depth of the pagination, as the database uses an index on the 'id' column to find the starting point. This approach ensures consistency, as new records inserted between requests won't cause duplicates or skips. It's particularly well-suited for infinite scroll and real-time data scenarios.

However, cursor pagination has its limitations. Clients cannot directly jump to a specific page number, and the encoding of the cursor adds some complexity. Most importantly, it requires a stable and unique sort key, typically the 'id' or a timestamp column.

Keyset pagination is akin to cursor pagination but uses actual column values rather than an opaque token. It's often employed with composite keys, consisting of multiple columns. The source provides a sample implementation of keyset pagination in Node.js. Keyset pagination is very efficient, utilizing composite indexes for optimal performance.

The main advantage over cursor pagination is that it doesn't require encoding and decoding of the cursor value, making it transparent and easier to debug. However, keyset pagination does require clients to understand the sort columns used, which may expose implementation details. It's also more complex to implement correctly, especially when dealing with multiple sort fields.

When choosing a pagination strategy, consider the specific requirements of your API. For admin panels or small datasets where page numbers are sufficient, offset pagination may suffice. For most public APIs, especially those dealing with frequently changing data or feeds, cursor pagination is the recommended default. Keyset pagination is ideal when maximum performance and control are necessary, and you're comfortable exposing sort fields in the API response.

Regardless of the pagination method chosen, it's essential to implement best practices. Cap the maximum page size to prevent abuse, such as limiting the limit parameter to 100 records. Include a next link in the API response so clients don't have to construct URLs themselves. Always document your pagination strategy clearly, specifying the required query parameters and the structure of the response.

A well-defined pagination mechanism ensures that clients can effectively consume your API data, regardless of the data volume.