The scourge of x86 emulation

This article explores the challenges associated with emulating the x86 memory model, specifically the Total Store Ordering (TSO), when running on ARM processors that utilize a weak memory model. The article explains that memory models define rules for how memory accesses behave in single-threaded and multi-threaded environments. The two primary memory models discussed are ARM's relaxed consistency model and x86's strict Total-Store-Ordering consistency model.

The TSO model guarantees that when a memory store occurs, it is coherently visible to all other processors in the system. Similarly, when a memory load occurs, all preceding stores are considered to have been completed. This model aligns with programmers' expectations, as writes become visible as soon as they are made, and loads reflect all previous writes. This intuitive behavior is a result of the model ordering loads based on the order of writes.

On the other hand, the weak ARM memory model allows for more efficient operation by not requiring immediate coherence across processors. This allows the CPU to prioritize power and efficiency by avoiding the hardware costs associated with invalidating or snooping cachelines. When a store instruction is executed, the affected cacheline is not immediately visible to other processors. Likewise, when a load instruction is executed, it does not guarantee that the updated memory value will be seen by other processors.

To address the memory model differences, ARM introduced load-acquire and store-release instructions in ARMv8.0-a. These instructions force memory ordering between affected instructions, similar to the std::atomic's memory_order_acquire and memory_order_release definitions in C++. By converting x86 memory loads to ARM's load-acquire instructions and x86 memory stores to store-release instructions, FEX achieves memory semantics that more closely match x86's behavior, albeit at a significant performance cost.

The article notes that emulating TSO using these instructions is exceedingly costly, with microbenchmarks demonstrating this fact.