Presentation: Instrumentation at Scale: Having Your Performance Cake and Eating It Too

Brian Martin, co-founder of IOP Systems, delivered a presentation on "Instrumentation at Scale: Having Your Performance Cake and Eating It Too" at QCon San Francisco. His work at Twitter and now at IOP Systems focuses on performance optimization and distributed systems.

He discussed the real-world performance costs of metrics libraries and shared strategies for low-overhead, fearless instrumentation. Martin explained that software is crucial in shaping the world and QCon is a practitioner-driven conference for technical team leads, architects, engineering directors, and project managers.

His talk focused on instrumentation, its importance, and the costs associated with it. He mentioned that instrumentation matters because it provides visibility into systems, helping to understand performance issues and diagnose production incidents. Without proper instrumentation, monitoring and improving performance become difficult.

Martin introduced metriken, a low-overhead metrics library he works on, which is optimized for performance-critical paths. It was developed for RPC-Perf and Pelikan, and later integrated into Rezolus, an eBPF-based systems performance telemetry agent.

He highlighted the impact of instrumentation costs, explaining that counter increments can range from 5 nanoseconds for uncontended atomic operations to over 1 microsecond for fully contended implementations. Histograms have an even wider cost spectrum, ranging from 7 nanoseconds to double-digit microseconds when fully contended.

The presentation aimed to provide insights into the cost spectrum of instrumentation, showcase real libraries, and introduce techniques for what Martin calls "fearless instrumentation." He emphasized that there is a need for low-overhead, performance-friendly instrumentation methods to maintain system visibility without sacrificing performance.