Flat vs segmented memory -- it's recursive

In recent years, the evolution of x86 architecture has shown a decline in fine-grained memory protection, which was once a significant feature of the system. This trend is partly attributed to the dominance of Unix, which prefers flat address spaces over segmented ones. This preference for flat memory is evident in the rise of technologies like WebAssembly, which shares similarities with OS/2 or other non-Unix operating systems where flatness is favored. However, this flatness raises safety and security concerns.

Poul-Henning Kamp, a well-known figure in the field, has proposed CHERI (Capability-Enhanced Reference Implementation) as a solution to these issues. CHERI addresses the limitations of flat memory models by reintroducing segmentation, which he argues provides better safety and security. However, the question arises: to what extent should hardware be involved in this subdivision?

The answer is recursive – programmers naturally subdivide the flat memory space repeatedly, creating a hierarchical structure. This recursive subdivision is a fundamental aspect of software development, where addresses are derived and passed around explicitly.

The non-recursive nature of hardware poses a challenge, as it is designed with finite-state machines and prefers fixed structures and bounded depths. Hardware can only handle a limited number of levels of decomposition, usually set to N=1. However, CHERI tackles this issue by allowing software to handle the recursive steps, with bounds that can be narrowed down by the software. This approach keeps hardware flexible and enables the software to address the memory based on the transitive closure of reachable capabilities.

While CHERI introduces some restrictions to maintain consistency and prevent misbehavior, it ultimately provides a more flexible and secure approach to memory protection. By capturing the recursive nature of software and its subdivision of the flat address space, CHERI aims to overcome the friction between hardware and software, ultimately leading to a more efficient and secure computing environment.