Core dump epidemiology: fixing an 18-year-old bug

OpenAI resolved apparently contradictory crash evidence by sorting a year's crash snapshots into distinct populations. Its engineering account says one group traced to a single physical host, while another exposed an old defect in GNU libunwind, software that helps C++ recover from exceptions.

The software defect involved timing: the library changed the active stack before finishing reads from its old location, allowing an arriving signal to overwrite data still needed. Engineers reproduced the problem and contributed a fix. ChatGPT helped write the script that classified crashes; the investigation depended on that cleaner dataset and subsequent human diagnosis. Separating the populations removed misleading counterexamples that had made a single explanation seem impossible.

“What we assumed was one problem eventually turned out to be two unrelated bugs, coincidentally discovered at the same time. First, silent hardware corruption on one Azure host, where the CPU just didn’t do math correctly. Second, an 18-year-old race condition in GNU libunwind, an unnoticed bug in a widely used open source library.”

Nathan Bronson

“In this case the vulnerable window is literally one instruction wide! A signal must be delivered after %rsp has been changed, but before the next instruction loads %rip . Several simple instructions like this can be run per cycle on a modern super-scalar out-of-order CPU, so the race window is roughly a hundred picoseconds.”

Nathan Bronson

“The most important step was not the clever assembly reading or deep knowledge of the details. It was building a high-quality data set.”

Nathan Bronson

“Once we had accurate and complete population data, the structure of the problem became obvious: one crash population belonged to a bad host, and the other belonged to a race in libunwind. Once the data got better, the debugging got easier.”

Nathan Bronson

“The GNU libunwind bug is old—more than 18 years old, present in the first x86_64 version that supported C++ exception unwinding.”

Nathan Bronson