Calling a function in C without naming it

In a remote code execution environment used for automatic code grading, a team sought a way to execute shell code without naming the execve function, as it is not permitted by the grading system. They hypothesized that by obtaining the address of execve in memory, they could call it directly. Modern executables employ Address Space Layout Randomization (ASLR), which randomizes the location of functions in memory, making direct address usage ineffective.

To bypass this, the team tried to determine a fixed offset between two known functions within the same memory segment, allowing them to calculate the address of the target function. They aimed to leak this offset by using a different symbol that is allowed, such as printf, which lives in the same segment as execve. However, they encountered issues due to Address Sanitizer (ASan) altering the behavior of functions like printf.

They considered an alternative approach using the mmap function, which allows creating memory segments with specific permissions. By leaking the offset between printf and mmap, they could allocate memory with both read and execute permissions, effectively circumventing the W^X security policy. The team then sought to modify memory containing unoptimized stack slots and employ unions to execute a custom x86 assembly routine that could make Linux system calls, enabling them to call any syscall with up to three arguments.