Tail sampling rules in the OpenTelemetry Collector: cost and pitfalls
Your second-tier Collector keeps restarting, or your trace bill refuses to fall: either way, it is your tail sampling rules that need another look, well before you touch the traffic. Every rule you add changes what the processor holds in memory, what it discards and what reaches the backend. This is not a configuration detail. It is a budget you have to arbitrate. This article starts from the…
This article explains the importance of tail sampling rules in the OpenTelemetry Collector, and how they can impact memory usage and performance. The tail sampling processor holds spans in memory until a decision is made, which means the number and size of rules directly affect memory consumption. The article provides a step-by-step guide for sizing and tuning these rules, using memory and throughput measurements to determine the appropriate values for decision_wait, num_traces, and maximum_trace_size_bytes.
Three types of sampling rules are discussed: exclusions, safety nets, and the cost ceiling. Exclusions drop noise spans, safety nets keep important traces, and the cost ceiling limits the number of spans per second. The article also covers combining rules and the nuances of using the probabilistic policy, as well as potential production pitfalls such as late spans and the impact of multiple Collector instances.
Lastly, the article lists key metrics to monitor and suggests re-reading the documentation with each Collector upgrade to ensure the tail sampling rules continue to function as intended.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.