Well I've never thought I was a member of an elite group because I was a pilot, but thinking about it, you are right. I wonder what percentage of the population we amateur pilots are? I must admit that most folk I meet at dinner parties are mightily astonished when they find out I used to fly gliders, and how much solo flying I had actually done.
On the other hand,it's also surprising how many folk have had a joy ride in a glider - maybe it is seen as an alternative to bungee jumping. Proves you have nerves of steel.