As you gain experience, that problem will fade out. Some places really demand respect, due to weather and turbulence, and you should consider that concern as a healthy response. You should not expect to be safe as close to a rugged mountain as you might be to a smooth sand dune - the air is very different between the two. For some places, I want 200 yards (meters) or more of open air between the mountain and me. I do not relax after launch until I am safely away from rugged terrain.
Also, oxygen can be an issue. Moving and flying at high altitudes can cause headaches, dizziness, nausea, confusion, and assorted mental issues. If you know your limits are about 10000 feet (930m) MSL, feel free to don the oxygen gear when near that limit, even when riding to launch or (especially) when setting up the glider. People who live at high altitudes may be okay there, but that may not be true for you, if you have been living at very low altitudes. A vague sense of dread or lack of confidence can be one sign of oxygen deficiency. It takes a few weeks or months to get a low-land pilot acclimated to the thin air of the high plains and higher mountains, but it will happen. Some sort of reasonable (gentle) exercise program can help, as will a healthy diet.
Hang tough. If you have concerns, it is healthy to discuss them here, or with experienced local pilots. We all share your dream.