My first blog post of the new year: sharing a huge pitfall one of our hardware teammates ran into. This might be the most magical bug I have ever seen, because it eerily exhibits a property reminiscent of the uncertainty principle in quantum mechanics: it is impossible to both reproduce the bug and observe the abnormal effect caused by the bug at the same time.
First, some technical background. Our team’s main work is writing numerical-computation logic for FPGA to accelerate compute-intensive workloads (the hardware side), and then writing the corresponding software drivers and APIs for those hardware functions and integrating them into our customers’ models (the software side).
From the software perspective, using FPGA for compute acceleration roughly follows this process: the hardware connects to the host (Host side) via PCI-E, maps its command registers onto a small contiguous region of physical memory on the Host side, so that when the Host reads/writes data at those memory addresses, it is effectively sending commands to the FPGA. The hardware then performs the corresponding computation based on those commands, and finally copies the results back to the Host to complete the operation. Taking matrix multiplication as an example, the process is:
- Matrix data is originally stored in the Host-side DDR memory, and is transferred via DMA (Direct Memory Access) to the DDR memory on the FPGA card;
- The Host writes data to the corresponding memory address (i.e., the FPGA’s command registers) (essentially executing a mov), thereby sending a compute command to the FPGA;
- The FPGA hardware logic starts executing the computation, and of course this process may take some time;
- After the computation finishes, the FPGA raises a hardware interrupt to inform the Host that the computation is complete;
- The Host DMA-copies the result from the FPGA back into its own DDR memory, thereby completing the computation.
Obviously, the compute time in step 3 has an upper bound. If the Host finds that the FPGA does not return an interrupt for a long time to indicate completion, the Host will consider the hardware faulty and trigger a timeout error—one of the scenarios hardware folks fear the most.
One day, during stress testing of the hardware logic, we found that under sufficiently high multithreaded load (more than three threads running at 100% CPU utilization while continuously sending commands to the FPGA), a very stable command-execution timeout occurred, while everything was normal in the single-threaded case. After a week of hard work, our hardware teammate checked all newly added logic and finally discovered that the real cause was a resource-contention bug in an old hardware logic module that had previously been running fine.
The details are roughly as follows. As mentioned earlier, before launching a computation, we first transfer the required data to the FPGA via DMA. All commands, including DMA, depend on the memory management unit (MMU) on the FPGA during execution. It is protected by a kind of “lock” mechanism in hardware logic: only the command logic that acquires this lock can proceed. However, because DMA commands execute very fast, in that old module, before the lock was acquired, the hardware mistakenly raised an interrupt to the Host, telling the driver that the DMA had completed and that it could send the next command. This is where the bug comes from. In reality, at that moment the hardware logic still cannot guarantee that the current DMA transfer has fully completed, and it is possible for this command to be preempted by the subsequent compute command. Then, when the Host’s compute command arrives, the computation uses data that has not yet been updated, naturally producing incorrect results.
Analogous to software engineering, this hardware bug is a typical multithreaded resource race. What is special about FPGA is that its computations are based on hardware logic: physically, if the electrical signal of the next command is to “overtake” the electrical signal of the current command before it finishes propagating, the time interval between the two commands must be extremely short. Therefore, this kind of command preemption only appears under very high pressure. But when we do stress testing, we can obviously only run the same compute logic on the CPU and compare the results to check correctness. The CPU is definitely slower than the FPGA (otherwise why would we bother), and with the huge data volume in stress tests, it is also impossible to store everything and check slowly (and storage itself also takes time). So as long as we try to verify the correctness of the results, we cannot put enough pressure on the FPGA to trigger the bug, and the computed results are completely correct; only by not observing it (i.e., not checking correctness) does it become possible to trigger computation errors.
It seems like such a bug could never be found, but how did we end up finding it? The hint was already mentioned above: we did not find incorrect results; we found timeouts. This happened because the newly added hardware logic, when faced with incorrect data, would execute far too many times (for example, we intended the hardware to execute a for loop 100 times, but with wrong data it actually tried to execute 1,000,000 times), leading to a timeout because it could not finish. That ultimately led our investigation here. As for why there is no timeout in the single-threaded case, that involves some driver implementation details, so I will not elaborate here. In short, if not for the new feature, this bug might have continued to hide silently in our hardware logic for who knows how long.
Some readers might ask: if such a data-computation error were triggered during real customer usage, wouldn’t the upper-layer application notice and directly manifest an error? This is because this compute acceleration system is used for training deep neural networks, and due to the well-known “alchemy” nature of DL, its robustness is sufficient to withstand such random errors without causing training error to blow up. It is even possible that the hardware randomness introduced this way somewhat improves training (because such a small bug might act as a kind of regularization, preventing overfitting), so...
That is the full story of this magical bug. In summary, the hardest bugs to debug are often tied to physical properties of the real world, thereby introducing boundless uncertainty, and they are difficult to reproduce and even difficult to notice.