OpenAI engineers recently resolved a series of elusive, long-standing system crashes in Rockset, the C++ data infrastructure service supporting ChatGPT plugins, by identifying a 18-year-old race condition in the GNU libunwind library. The investigation revealed that the failures were not a single issue, but two distinct syndromes: one caused by faulty hardware and another by a 100-picosecond race window during exception unwinding.
Distinguishing Hardware Failure from Software Defects
The OpenAI team initially struggled to diagnose the Rockset crashes because the symptoms were inconsistent. According to the team’s engineering report, they lacked a clear data set, causing them to conflate two different types of failures. By building a script to automate the analysis of core dumps, they discovered that what appeared to be one issue was actually two.
One subset of crashes was traced to a specific Azure data center host. This hardware was silently producing incorrect mathematical results without triggering standard machine-check exceptions or overheating. Once the team identified this “corrupted” cluster, they removed the host from service, effectively eliminating that category of crash.
The race condition discovered in GNU libunwind had persisted for 18 years before being identified by the OpenAI team. It went unnoticed in most applications because the timing window required to trigger it—roughly 100 picoseconds—is incredibly narrow.
The 100-Picosecond Race Condition
Once the hardware-related crashes were filtered out, the remaining “return-to-null” crashes became easier to analyze. The root cause was located in the _Ux86_64_setcontext function within GNU libunwind. During C++ exception unwinding, the function updates the stack pointer before finishing its read of the instruction pointer.
This creates a 100-picosecond window where, if a signal is delivered to the process, the kernel overwrites the stack frame with signal data. This corrupts the instruction pointer, leading to a crash. While most applications rarely trigger this, Rockset’s use of timer_create to deliver frequent SIGUSR2 signals for per-query accounting made the race condition a production reality. The team has since upstreamed a fix to the GNU project that reorders these instructions to ensure safety.
Lessons for Future Debugging at Scale
The OpenAI investigation offers a blueprint for teams managing massive, high-throughput data systems. The primary takeaway, as noted by the engineers, is that the most critical step was not deep assembly knowledge, but the creation of a high-quality, labeled data set. When individual cases resist explanation, engineers should consider that they may be conflating multiple, separate phenomena.
If your service is experiencing crashes that seem to defy logic, stop debugging individual instances. Instead, build a classification tool to aggregate all available crash data. You may find that your “unsolvable” bug is actually two solvable ones.
Frequently Asked Questions
Why did this bug take 18 years to find?
The race window is only one instruction wide, or about 100 picoseconds. Under normal circumstances, the probability of a signal arriving in that exact fraction of time is near zero. It required a specific, high-frequency signal environment—like the one created by Rockset’s accounting system—to make the bug reproducible.
Is this a common issue in C++ development?
While race conditions are common, this specific issue was unique to the implementation of libunwind. The team verified that other unwinders, such as libgcc, do not share this specific vulnerability.
What should teams do if they suspect silent hardware failure?
The OpenAI team’s experience suggests that if a specific subset of errors originates from a single cluster or host, isolating those nodes is the fastest way to confirm hardware interference. If the crashes stop after removing the hardware, you have found your root cause.
Have you encountered an “impossible” bug that turned out to be two separate issues? Share your experience in the comments below or subscribe to our engineering newsletter for more deep dives into production system reliability.
