My most memorable hardware bug was noware near as hard as this, but I'll never forget it.
Intel was trying to sell the 960s and sent us a dev board with that CPU. Nobody in the company could get it to boot up. It would power up but nothing would show up on the serial port. Eventually it was my turn to look and for some reason I happened to notice a pullup capacitor on the UART VCC. I looked at the schematics and indeed it was there. A simple jumper to bypass it (back in those days we had big, manly components; none of that surface mount shit) and what hey: the serial console responded. It had booted up just fine, but was mute.
After that we could do development but it was immediately clear to me that the 960 was DoA. It's not like we were the first to get that board!
I was debugging a TI DSP based board that I designed. It would come up, execute some of my code, and die. It took 3 weeks and lots of back and forth with TIs tech support before I found out that some of the ground pins were left disconnected. The guy that laid out the PCB didn't connect them though they were connected in the schematic. We went back to the PCB editor, zoomed way in, and lo and behold there was this tiny unconnected segment.
I don't remember the details any more (this was like 30+ years ago) but I think TI's support was on the right track and I was convinced this can't be true because I had the connection in the schematic. If you probe this with a DVM or a scope it will look connected (because the pins are connected internally) and so it's really hard to find out.
This taught me a valuable lesson that anything can be debugged with enough time and persistence. Some things take longer and that's life.
Fucking floating pins, man. They're the worst, because they work during bringup and development, and only fuck you over when scaling to mass production
Instead of a pull-up resistor between the uart vcc pin and vcc, there was an electrolytic cap. That is why I put “capacitor” in italics — I was trying to emphasize how nonsensical it was. To the DC power to the chip the cap was of course an open circuit and thus by shorting it out I powered the chip up.
Intel was trying to sell the 960s and sent us a dev board with that CPU. Nobody in the company could get it to boot up. It would power up but nothing would show up on the serial port. Eventually it was my turn to look and for some reason I happened to notice a pullup capacitor on the UART VCC. I looked at the schematics and indeed it was there. A simple jumper to bypass it (back in those days we had big, manly components; none of that surface mount shit) and what hey: the serial console responded. It had booted up just fine, but was mute.
After that we could do development but it was immediately clear to me that the 960 was DoA. It's not like we were the first to get that board!