I've been thinking about your post for quite a bit now, and have had long debates in my head about how to go about it all.
First off, I don't think postponing things and having a bit of a late start is too big a problem, especially not in our part of the world (well, I hope it's not or else I'd be in deep trouble!). I can see how your anxiety is an issue in this case and I don't think it's something that should be belittled at all. I've got a mood disorder myself that I accidentally became really aware of when I started getting into Buddhism, and meditation and the Dharma has helped immensely. As a matter of fact I probably wouldn't be doing very well at all without it. There are still things I wouldn't subject myself to, though. I know I'd never be well off in some kind of student house, I know I need to make sure I can sleep at night and not listen to a neighbour's loud music through the wall etc. I know I need to maintain some kind of stability in the world. You should find out what you need to do and have to deal with your anxiety. Sure, it's good to push yourself sometimes, see if things are really as you think they are, but if you have something like anxiety or something more serious then remember to take care of yourself, respect yourself and your boundaries. If you want to go to school or uni make sure you research things and find a place that suits you well, including your living situation during your education.
Another thing is probably to switch your focus a bit. Ask not what the Dharma can do for you, but what you can do for the Dharma! I found out last year that a girl I went to school with as a child had gotten really into Thich Nhat Hanh, Plum Village, and all that jazz. She'd gotten herself a degree in social education - and spent time working with child monks in the East! She seems to be very involved with the tradition, and is doing hers to be an active part. And what a brilliant way of doing it! Other people in the thread have talked of things similar to this. Of finding something you can do, a qualification that can help you explore the world and the Dharma in different ways.
You don't have to pick between the Dharma and a Western life, but I can totally follow how you feel about all this. This world doesn't go easy on people of a more sensitive nature, you just have to find a way to work around it. You can always look into the whole Off The Grid movement as well. Maybe spend a summer in the woods in Scandinavia somewhere. Get a bit of distance between yourself and the busy cities.