Non-Determinism and Overcount on Modern Hardware Performance Counter Implementations

Ideal hardware performance counters are expected to provide exact, deterministic results. However, this is not always the case with real-world performance monitoring unit (PMU) implementations on x86_64 machines. Events that should be deterministic, like retired instructions, exhibit variation and overcounting even in strictly controlled environments. These non-intuitive effects pose challenges for applications requiring strict determinism, such as deterministic replay or deterministic threading libraries.

A study was conducted on eleven different x86_64 CPU implementations to investigate the sources of divergence from expected count totals. Only a few counter events were found to have enough determinism to be used without adjustment in deterministic execution environments. Notably, the iret interrupt return instruction appears to count as a userspace instruction despite running in kernel space, leading to non-deterministic counts.

Counter events that do not count for iret, such as conditional branches and retired stores, appear to be more deterministic.

The investigation also extended to ARM, IA64, POWER, and SPARC systems, where the counter events exhibited more determinism. The study explores various methods of working around the limitations of x86_64 events, but in many cases, this would require architectural redesign of the underlying PMU.