Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why
How we reduced Kafka stream volume by 80%, cut Spark workers from 240 to 30, and lowered reporting costs through sampling and executor tuning at Unity.
At Unity, the ad exchange recorded every phase of an auction process in Kafka, which generated approximately 10 GB of data per second. The challenge was that each downstream Spark job processed the entire volume, despite needing only a fraction of it. The author, a data engineer, proposed sampling the stream and suggested changes to the Spark clusters.
The project aimed to reduce the cost of storing and processing the same large data volume at various stages, such as Kafka storage, object storage, Spark batch jobs, and Druid indexing. Most of the data was not needed by the reports, with only 1% of the data volume being relevant. The author mapped the events in the stream against the reporting requirements and found that about 99% of the data came from bid requests filtered out before the auction, while the reports relied on post-filter events like bid responses, auction decisions, fills, impressions, and clicks.
To reduce the data volume, the author decided to pre-filter events at a 0.5% rate and retain 100% of post-filter events. They used the auction ID as the sampling key to maintain relationships between related events. After implementing the changes, the data volume dropped from 10 GB per second to 2 GB per second. The author then optimized the Spark clusters, reducing CPU utilization by 41% and cutting the number of worker nodes from 240 to 30.
They also discovered and fixed an issue in the BidWin and WinPrice metrics, which had been calculated incorrectly since 2019 due to a join causing row duplication.
Written by urgent.news from HackerNoon's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
