Word 97's ghost in the machine: The crash that vanished the moment anyone looked at it

Veteran Microsoft engineer Raymond Chen shared another fireside tale – one to strike fear into the hearts of engineers everywhere: the bug that can't be replicated in the debugger. The story begins near the end of the Word 97 release cycle, when a defect surfaced in the test labs but vanished the moment engineers reached for a debugger. It was maddening. A script could trigger the crash fairly reliably, yet the instant anyone tried to hunt it down, the bug ducked out of sight. Chen recalled: "The development team was very anxious, because a bug that appears only sporadically in the lab is going to occur regularly in the wild. But how do you debug a problem that resists debugging?" It is every developer's nightmare, even for those smug types who lean back in their chairs and tell testers "works alright here, mate." There is a special place reserved in the afterlife for these developers. With the bug refusing to cooperate, talk turned to an ICE (an In-Circuit Emulator, not the immigration agency), which could stand in for the CPU and allow breakpoints inside the emulated chip itself. "To most software developers," said Chen, "in-circuit emulators existed only in myth, and the possibility of acquiring one and using it was met with the excitement of a five-year-old boy who learns that he might get to ride on a fire truck." Eventually the team traced the problem to a specific manufacturer, and from there to a CPU erratum. The compiler in use at the time had been updated to dodge the problematic code sequence. However, by then the toolset had already been locked. Swapping in a new compiler risked introducing fresh, unknown bugs, so that was off the table. The solution? Scour the binaries to find instances of the code that might upset CPUs. One was found. "To avoid introducing any new problems as a side effect of the fix, the team opted to do a binary patch to the binary to insert a nop into the offending code sequence." Word 97 introduced plenty of headline features. There was the arrival of Clippit (aka Clippy) and Visual Basic for Applications (VBA). And then there was a binary patch produced by determined engineers, which received far less attention. But without it, Word 97 could have been distinctly wobbly. ®