It shows how to apply epidemiology-style, large-scale core dump aggregation to isolate rare crashes across a fleet — a debugging approach that separates a hardware fault from a decades-old software bug when single-instance debugging fails.
Key quotes
“What we assumed was one problem eventually turned out to be two unrelated bugs, coincidentally discovered at the same time. First, silent hardware corruption on one Azure host, where the CPU just didn’t do math correctly. Second, an 18-year-old race condition in GNU libunwind, an unnoticed bug in a widely used open source library.”
Nathan Bronson
“In this case the vulnerable window is literally one instruction wide! A signal must be delivered after %rsp has been changed, but before the next instruction loads %rip . Several simple instructions like this can be run per cycle on a modern super-scalar out-of-order CPU, so the race window is roughly a hundred picoseconds.”
Nathan Bronson
“The most important step was not the clever assembly reading or deep knowledge of the details. It was building a high-quality data set.”
Nathan Bronson
“Once we had accurate and complete population data, the structure of the problem became obvious: one crash population belonged to a bad host, and the other belonged to a race in libunwind. Once the data got better, the debugging got easier.”
Nathan Bronson
“The GNU libunwind bug is old—more than 18 years old, present in the first x86_64 version that supported C++ exception unwinding.”
