Indirect Calling of Nested Functions on GCC Without Executable Stack

In discussions about calling nested functions in GCC and Clang, it was noted that older versions of GCC can be supported without requiring an executable stack. While enabling an executable stack is not ideal, a workaround exists. Traditionally, GCC supports taking the address of a nested function, as demonstrated in a toy example.

By analyzing the trampoline code, it is possible to extract the code address and static chain, which can then be used with the __builtin_call_with_static_chain built-in function to call the local function directly. This method can be utilized as a fallback mechanism in older GCC versions.

The main drawbacks of this approach are the creation of a trampoline, the inability for the compiler to devirtualize the indirect call, and the fact that the stack remains marked as executable. However, the security concern of an executable stack can be addressed using patchelf --clear-execstack. Another idea proposes using the trampoline itself as a function descriptor, allowing the code of the trampoline to be interpreted at the call site using a simple interpreter.

This technique could potentially be implemented in an experimental library like noplate, where a wide pointer is constructed from the code address and static chain.