This is what I was going to say - suspect this stems from having parents or guardians who cared enough to get you involved in such an activity, and the environment that suggests, rather than scouting directly.
There's a lot to be said for going for a 3-day hike in the wilderness, even if you're accompanied by a parent.
Going with a scout group usually means that you wont be relying on the parent as you want to stay as a member of the group of people your own age.
You'll start to feel tired and sore at some stage, and it may start to rain. But if you've prepared properly (and the scout motto is "be prepared"), then you should be ok.
It's just a matter of getting away from it all, and perhaps embracing the suck as mentioned above.
It's true - but I scouted for years and still turned out neurotic and anxious - but I had dismissive parents who I only spent a few months or so with throughout my childhood.
