The scourge of x86 emulation

In this feature article, we will explore the ongoing issue with x86 emulation and its impact on applications that rely on emulation. The core problem lies in the total store ordering memory model (x86-TSO) and the challenges it presents when emulating it on systems that use a weaker ordering memory model like ARM's. This discrepancy can lead to a variety of issues and complications that we will delve into throughout the article.

Before we discuss how to address the x86 memory model problem, it is crucial to understand what a memory model entails. A memory model defines the rules that govern how memory accesses behave within a system, whether it be single-threaded or multi-threaded. There are several popular memory models utilized in different hardware forms, with ARM's relaxed (or weak) consistency model and x86's Total-Store-Ordering consistency model being the primary focus of this discussion.

The x86-TSO model is highly strict, enforcing a strong coherency model that doesn't permit much room for optimization. This means that when a store occurs, it is coherently visible to all other processors in the system, and when a load occurs, all prior stores are logically completed or at least visible. In contrast, ARM's weak memory model operates differently.

By default, ARM's regular memory loads and stores aren't strictly coherent across processors, allowing for more efficient hardware operation. When a store instruction executes, the affected memory (cacheline) isn't immediately visible to other processors. Similarly, when a processor loads data from memory that another processor has written to, it's not guaranteed to see the updated memory.

This can cause significant problems in multi-threaded applications. To bridge this gap, ARM introduced load-acquire and store-release memory instructions, which are not technically atomic operations in C++ parlance but are often conflated as such. These instructions force memory ordering between class operations. In ARM's "Release Consistency sequentially consistent (RCsc)" model, load-acquire instructions must be observed sequentially without reordering, while store-release instructions must as well, fulfilling barrier-ordered-before semantics.

To emulate the x86-TSO memory model in ARM, FEX converts x86 memory loads into ARM's load-acquire instructions and x86 memory stores into store-release instructions. This approach provides FEX with memory semantics similar to x86, albeit more strict than necessary due to the lack of a middle-ground solution. However, this emulation method is incredibly costly for the hardware, particularly when executing a higher proportion of acquire/release instructions than traditional ARM CPUs were designed for.