Using TPC-DS to Compare Cloud Data Engines by Price Performance

Cloud vendors often present benchmark results under conditions they have chosen, making it difficult to compare their engines objectively. The author needed a standardized, repeatable benchmark that could assess the actual performance and cost of various cloud data processing options for a large organization. TPC-DS (Transaction Processing Performance Council - Decision Support) was chosen because it is a neutral, analytical benchmark that uses a common data model and a set of standard SQL queries to measure performance across different engines.

By running the same workload on multiple systems and measuring the cost of performance, the author could compare the engines more fairly, without being influenced by vendor-specific optimizations or one-off tests. The benchmark revealed that performance depends on the workload size, with some engines performing well for small datasets and others excelling at larger sizes.

Serverless options were particularly attractive for smaller jobs due to their lower overhead. The author concluded that instead of standardizing on a single engine, organizations should determine which kinds of workloads are best suited for each engine, allowing them to optimize costs and performance based on the specific needs of their data pipelines.

However, the author cautioned that synthetic benchmarks like TPC-DS may not fully capture the complexities of real-world data, such as skew, distribution, schema changes, and historical baggage. They also noted that the economic benefits of moving workloads to more elastic processing architectures may not be immediately apparent, as caching and concurrency can further influence the performance and cost of analytical workloads.