Might be because I never owned one myself, but I've always felt the interface incredibly clunky and the navigation hard to grasp. It's not something intuitive you know how to use just by looking at it.
That's kinda my point, though. You couldn't just pick up the iPod connected to the speakers at a friends' place and instantly know how to switch songs. While virtually every other mp3 at the time had a d-pad with labeled buttons easily understood.
I think OP’s right, everyone “knew” how to operate an iPod through advertisement and probably few had to figure it out on their own. It was simple and consistent enough that they looked intuitive but maybe it wasn’t.