Design an ad click aggregator: the system design interview walkthrough

Design an ad click aggregator: the system design interview walkthrough

Designing an ad click aggregator appears simple at first glance – tallying the clicks on each ad. However, this one-line request conceals a complex system design interview scenario. The interviewer introduces a real-time analytics challenge, akin to Google Ads, where tens of thousands of clicks occur every second and advertisers demand live updates on clicks-per-minute for their campaigns. At this scale, scanning billions of raw rows per query or maintaining a precise running total becomes impractical.

The core of the problem lies in the firehose of writes vastly outpacing the trickle of reads. Thus, the design revolves around constructing a streaming aggregation pipeline capable of ingesting an unbroken flow of clicks without loss and transforming them into queryable numbers. This pipeline begins with buffering the continuous stream of clicks in a log, followed by pre-aggregation in a stream processor over defined time windows, and finally serving these pre-computed roll-ups (clicks per ad per minute) from an analytics store for quick queries.

A crucial aspect of this pipeline is the billing requirement – the counted clicks represent financial transactions, necessitating both speed and accuracy. Therefore, a secondary path is introduced for an authoritative, slower batch computation that reconciles the pre-aggregated counts from the raw click events. This dual-layer architecture embodies the Lambda architecture, combining a fast-speed layer with a correct batch layer to ensure billing accuracy.

The interview progresses through several structured steps, each emphasizing a different facet of system design. Initially, clarifying requirements – both functional (recording clicks, providing aggregated metrics, and enabling real-time dashboards) and non-functional (extremely high write throughput without data loss, low-latency reads, horizontal scalability, and resilience) – sets the foundation.

An estimate follows, comparing the scale of incoming clicks to the rate at which advertisers query aggregate data to justify subsequent design decisions.

The high-level design step involves sketching the system's architecture, typically comprising four primary components: a log for ingesting clicks, a stream processor for aggregation, a service layer for serving roll-ups, and a batch job for reconciliation. Subsequent deep dives focus on critical decisions like choosing a partitioned, retained log; implementing an event-time windowed aggregation with watermarks in the stream processor; ensuring idempotent writes; and architecting a reconciling batch job.

The interviewer probes these areas to distinguish between memorized answers and genuine comprehension of the underlying principles.

With a tight 45-minute timeframe, the focus shifts to executing the ingestion, aggregation, and accuracy components without delving into ad-serving complexities. The interviewer frequently interrupts to probe deeper into each decision, ensuring the candidate demonstrates not just theoretical knowledge but practical understanding of high-write pipelines.

The debrief then evaluates the candidate's performance across four dimensions: problem-solving (reframing the challenge as a scaling-writes scenario and emphasizing the log-aggregator-read structure), communication (clearly articulating the rationale behind buffering the firehose, time-bucketing, and the dual-layer reconciliation approach), technical execution (detailing the chosen technologies like partitioned, retained logs; event-time windowed aggregations with watermarks; and idempotent writes), and time management (allocating the 45 minutes efficiently to core system components).

Ultimately, the debrief assesses these dimensions rather than the correctness of the design, providing a nuanced measure of the candidate's ability to navigate complex system design challenges under pressure.