The caveat is not to overdo it though. Learning calculus from not-the-best book is still way better then dreaming around and procrastinating on which textbook really is the very best. (Same with workouts, diets etc.)

And in terms of popular non-fiction books, like Harari or so, they are best thought of as textual Rorschach tests. He throws up topics and connectioms, activates certain parts of your mind and you can now fly from there. Even if you know a lot of background, many such books can be great thought stimulants.