S3 Storage Tiering: Pushing Cold Data Down the Ladder

S3 storage tiering is a cost-saving strategy that involves moving data between different storage classes based on its frequency of access. This process, also known as "pushing cold data down the ladder", allows organizations to store data in the most cost-effective manner without deleting any data. The storage classes, arranged from hottest to coldest, are Standard, Intelligent-Tiering, Standard-IA, Glacier Instant Retrieval, Glacier Flexible Retrieval, and Glacier Deep Archive.

The ideology behind S3 storage tiering is simple: each object should be placed in the cheapest storage class that can still accommodate its access latency needs. Over time, as the data cools, it moves down the ladder to cheaper storage classes. AWS S3 offers six storage classes, and sets of rules, known as lifecycle rules, are used to transition objects from one storage class to another.

The primary advantage of S3 storage tiering lies in its ability to automatically transition data to the most cost-effective storage class, thus reducing recurring storage costs. It is important to note that colder storage classes, such as Glacier Deep Archive, add a per GB retrieval charge and a minimum storage duration (30, 90, or 180 days). Therefore, the move only makes financial sense for data that is rarely accessed.

The main reason why cold data drains budgets is because most S3 bills are primarily composed of storage costs, not request charges. Even if an object is accessed infrequently, it continues to sit in the Standard class, accruing the highest per GB rate on the ladder daily. Until a lifecycle rule is applied, teams seldom delete or move this data, resulting in unnecessary waste.

S3 Intelligent-Tiering is an intelligent version of storage tiering that automatically moves data to the most cost-effective storage class based on its access frequency. Its Infrequent Access tier saves up to 40% on storage costs compared to Standard, the Archive Instant Access tier up to 68%, and the Deep Archive Access tier up to 95%.

Lifecycle rules are crucial in pushing objects down the S3 storage ladder. A lifecycle rule is an XML configuration attached to a bucket, specifying the transition action and the day count. AWS provides an example of a lifecycle rule for logs objects, which tiers the logs/ prefix over a lifetime. This rule moves objects to the IA storage class after 30 days, to Glacier after 90 days, and expiring them after 365 days.

To implement this rule, you can use the AWS CLI command `aws s3api put-bucket-lifecycle-configuration`. This command accepts a lifecycle configuration file as input. This approach works seamlessly with RustFS, an S3-compatible storage service.

When deciding between Standard and Standard-IA, the decision depends on the frequency of data reads. If an object is read more than about once a month, the retrieval fees incurred by Standard-IA negate its storage cost savings, and thus it should remain in Standard. Conversely, if an object is read quarterly or less frequently, IA becomes the more cost-effective choice. However, IA has a 128 KB minimum billable object size, so objects smaller than this threshold might incur additional overhead costs.

The three tiers of Glacier storage each have their unique retrieval times and minimum durations. Glacier Instant Retrieval offers millisecond access at a per GB retrieval fee, with a 90-day minimum duration. Glacier Flexible Retrieval (formerly S3 Glacier) provides minutes to hours of retrieval time and a 90-day minimum. Glacier Deep Archive is the slowest of the three, with hours-long retrieval times and a 180-day minimum.

The retrieval minimums are a significant consideration, as deleting or retiering an object prematurely incurs prorated fees. Consequently, Glacier storage classes are most effective for data with a true retention horizon, such as backups and compliance archives.