Urgent.News

What's breaking now, across thousands of outlets.

Tech

Validating a High-Performance Cloud Object Store with Lightweight Formal Methods

Abstract: We report on our experience validating S3 Express One Zone, a high-performance Amazon S3 storage class launched in November 2023. We focused on its distributed metadata index, which implements a hierarchical namespace using distributed transactions. We check strong consistency and durability under crashes, network delays, and timeouts. Building on our prior work on lightweight formal…

In November 2023, Amazon S3 launched a new high-performance storage class called S3 Express One Zone. Our focus was on its distributed metadata index, which manages a hierarchical namespace through distributed transactions. We examined strong consistency and durability in the face of crashes, network delays, and timeouts. Drawing from our prior work on lightweight formal methods, we integrated executable reference models with property-based testing and stateless model checking.

To apply these techniques to a real-world distributed system, we had to devise three novel methods. Firstly, a white-box approach was used to verify serializability. Secondly, we validated both the data and control planes in a compositional manner. Lastly, long-running canaries were employed, utilizing reference models to confirm correctness across different deployments.

We adapted Shuttle, our open-source stateless checker, to function effectively within a production async Rust runtime, and incorporated failure injection methods, including cancelled operations and timeouts.

Our primary conclusion is that this approach is practical and scalable at engineering scale. Following the S3 Express One Zone's launch, the engineering team took charge of the models and test infrastructure. Their share of validation commits rose from 41% to 84%, and they independently expanded the models to accommodate conditional writes, object append, and atomic rename features. This expansion occurred with minimal involvement from the formal methods team.

Written by urgent.news from Lobsters's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dl.acm.org →

More in Tech

Do not buy RAM, compress

Article URL: https://curiousbits.nl/blog/do-not-buy-ram-compress/ Comments URL: https://news.ycombinator.com/item?id=50557661 Points: 288 # Comments: 235

More from Thursday 19 November →