One new label multiplied our metrics by every order we take

A single label transformed the team's metrics exponentially with every order processed. This label, an order id, went through a review process in just four minutes due to its brevity and sound reasoning. On Sunday evening, the metrics server began restarting in a loop, leading to a complete lack of monitoring on Monday morning during a real payments incident.

Each label is a unique dimension, with its own time series and memory-resident index. With sixty thousand order ids generated daily, the number of series skyrocketed from one point one million to fourteen million in under three days. The failure was not clean, as ingestion slowed, alerts arrived late, and some evaluation cycles were skipped.

The system killed itself for memory, replayed its write-ahead log, and was killed again. To mitigate the issue, the label was removed from the scrape using a relabel rule. The guards were then deployed, with each service having a series budget and a rule that triggered at eighty percent of it. The incident was quickly identified through a panel showing the top metric names by series count.

The lesson learned was that while an identifier might seem reasonable, it should be placed on a trace or an exemplar attached to a bucket, as it multiplies across every dimension and incurs significant memory usage.