I am 35 and been flying almost a year now. Here's my hypothesis for what it's worth. I imagine hanggliding initially attracted a great number of thrill seekers and risk takers, it was genuinely almost suicidal, and while the sport has become substantially safer, that reputation stuck.
In the meantime crazier sports like base jumping, wing suits, and so on likely have attracted a good number of people who hanggliding would have attracted in the past.
While at the same time of course PG's with their lower cost of entry, convenience, and perceived safety have attracted many of the "I just love to fly" type of pilots. By the numbers the two are similar in danger, however PG never gained the reputation hanggliding did, and while skydiving and PG don't have much in common I think PG inherits much of skydiving's public perception because they look similar to the casual observer.
This leaves HG in a difficult spot, on one hand it doesn't tend to attract the suicidal thrill seekers, and on the other hand the general reputation for extreme danger puts people off before they bother trying it and PG tends to snap these people up.
As for my personal anecdote: I've wanted to fly since I was very young. When I lived in Phoenix I briefly looked into HG, casually mentioning it to some friends and family and got some pushback about how dangerous it was. I wasn't serious about it and so I let myself get talked out of it. 10 years ago or so I got my pilots license and flew a cessna 150 for awhile, I actually brought HG up with my flight medical doctor saying I wanted to try it and he pretty much said "Oh no, don't do that I know a bunch of people that got hurt badly hanggliding" he was in his 70's. Again, I wasn't attached to the idea and hadn't really looked into it at all, so again let myself be talked out of it. A little over a year ago now I was browsing around youtube and got to watching a bunch of hanggliding videos. Specifically Greg Porter's XC flight from Migus to Meteor crator, as well as one of a guy thermalling amidst clouds. Those video's hooked me, it was something I was going to do regardless of risk. A couple weeks later I was a H2 despite having to drive 4.5 hrs each way every weekend.
In my case It was HG instead of PG because for one I didn't know about PG, and other than that it kinda looks silly
So in short, I think HG's primary issue is the undeserved reputation that still sticks to it since the 70's. Everything Davis said is pretty much spot on, HG is local and young pilots are an anomaly, most start in their 30's-40's.
If I were going to start a flight park, I would advertise to GA pilots and skydivers. Most GA pilots get into it because they have always wanted to fly and it's the obvious way to go about it. However you have to make your own fun (100$ hamburger run, or meetup's etc), with hanggliding the fun is built in, and certainly for me resembles more of what my dreams are like than flying GA aircraft.