If there was a magic pot of gold, it would be technically possible to precompile every crate version with every rustc version on every supported platform and distrivute those prebuilt rlibs to users through cargo. That would help with first compile times when using the standard tooling, and not just for proc-macros.
Different people with different use cases have different complains. I haven't quantified but have certainly seen various complaints about both cases from different people.
