The scourge of x86 emulation

The x86 emulation problem is a significant issue affecting every application that we emulate. At the heart of this problem is the x86-TSO (Total Store Ordering) memory model. The emulation of this memory model on ARM's weak ordering memory model (ARM-TSO) presents multiple challenges. This article will explore these issues and the solutions that can be implemented.

Prior to discussing the solutions, it is essential to understand what the x86-TSO memory model entails. A memory model defines the rules governing memory access behavior in a system, both in single-threaded and multi-threaded environments. Two popular memory models in hardware are ARM's relaxed (or weak) consistency model and the x86 variant of Total-Store-Ordering consistency model.

The former is more relaxed, enabling significant hardware optimizations, while the latter is stricter, enforcing a strong coherency model that limits optimization opportunities. When discussing memory models, it is crucial to distinguish between consistency and atomicity, as they are related but not the same. ARM's weak consistency model allows for efficient hardware operation by permitting significant hardware optimizations, as stores are not immediately visible to other processors.

Similarly, x86's Total-Store-Ordering model enforces a strong coherency model, ensuring that memory stores and loads are sequentially consistent, making it easier for programmers to predict the behavior of their code. In contrast, ARM's weak memory model is less predictable, as stores are not immediately visible to other processors, leading to potential issues in multi-threaded applications.

To address these inconsistencies, ARM introduced load-acquire and store-release instructions, which map to C++'s std::atomic memory_order_acquire and memory_order_release definitions. These instructions help force memory ordering between specific classes of instructions, with ARM's model being called "Release Consistency sequentially consistent (RCsc)."

By converting x86 memory loads to ARM load-acquire instructions and memory stores to store-release instructions, FEX can emulate the x86-TSO memory model while maintaining its strictness. However, this emulation is highly resource-intensive, as evidenced by microbenchmarks showing significant performance impacts.