How we found a bug in Go's arm64 compiler

Cloudflare’s post details how a scale-driven bug hunt uncovered a race in Go’s arm64 compiler that affects stack unwinding. The team observed sporadic panics and fatal errors on arm64 machines, with crashes linked to unwinding the goroutine stack during garbage collection and error handling. Two crash classes emerged: a crash while accessing invalid memory and an explicitly checked fatal error, both tied to the unwinder reading a partially adjusted stack pointer.

The breakthrough came when a production coredump showed a preemption occurring between two stack pointer adjustments in Netlink’s Receive function, revealing a single-instruction race window in the compiler’s IR-to-machine-code generation. Once the root cause was confirmed, Cloudflare helped push a fix upstream, implemented in Go versions go1.23.12, go1.24.6 and go1.25.0, so that stack pointers remain valid during preemption.

The team emphasises that the bug was a rare, large-scale race requiring deep dives into the Go runtime and compiler internals, and they continue to invite collaboration from others who enjoy this kind of debugging detective work.