What it really boils down to is this: you cannot have both caching and privacy.

For the cache to work your browser must reveal to the server what it has already downloaded, this way or another. And the browser cannot really tell which of the downloaded pieces of data were specially generated to track this particular user.

A possible workaround is to create an intermediate cache to share it with multiple other people, but this creates other privacy concerns.