Hunting Down a Go Runtime Bug on 32-bit Embedded Systems

An application written in Go started crashing intermittently on a 32-bit ARM embedded Linux system. Initially, it appeared the application was at fault, but further investigation revealed the issue originated within the Go runtime itself. The fatal error stemmed from the netpoll mechanism, specifically the netpoll_epoll.go file.

The core problem was that Go's netpoll code expected only the EPOLLIN event, but instead received a combination of EPOLLIN and EPOLLOUT. The discrepancy arose because the epoll mechanism reported a firing event with both flags set, despite only EPOLLIN being requested. Upon searching for similar reports, the development team found a matching issue in the Go project's issue tracker, filed on March 2025.

The issue noted that the crash occurred on 32-bit ARM and i386 Linux systems and was triggered when the application ran for an extended period. Despite the issue being open and unresolved, multiple reporters experienced the same problem but no fix had been implemented. The team hypothesized that epoll might behave differently on 32-bit ARM or i386, but quickly dismissed this as the kernel core code should not exhibit such variations.

They then examined the epoll usage within Go's netpoll code, focusing on potential pitfalls on 32-bit platforms. The investigation revealed that Go's netpoll code utilizes the data field of the epoll_event structure, which on Linux can hold an 8-byte cookie in the kernel. This cookie is returned to user space and is used by applications to attach metadata to events.

Within the Go runtime, the netpoll logic compares ev.Data to the address of an internal event fd object to determine if an event belongs to an event fd or a socket fd. The comparison was found to be flawed as it only considered the lower 4 bytes of ev.Data, which on 32-bit platforms would lead to aliasing the address with the fdseq value, causing the netpoll logic to incorrectly identify socket fds as event fds.

This aliasing issue only manifested on 32-bit little endian systems, where the comparison would erroneously rely on the lower 4 bytes. The problem was exacerbated by long-running programs generating numerous pollDesc objects over time, with fdseq eventually aligning with &netpollEventFd after reaching approximately 3 million. To resolve the issue, the team proposed a fix that involved storing a nil pollDesc as a tagged pointer instead of the raw pointer to netpollEventFd.

When unpacking a tagged pointer, the value nil would indicate that the event belongs to the event fd, otherwise, it would signify a socket fd. This modification would eliminate the aliasing issue and prevent the crash from occurring.