I remember that space is completely unforgiving and we just aren't up to the task for anything more than a token selfie by the best dozen humans we can possibly produce with great effort and training.

Astronauts aren't superhumans and there is nothing "special" about their training. They are just pilots with stricter physical requirements. The reason why there aren't many of them is because there is no need for more. Our technology is not there yet for cheap and "boring" space travel beyond low Earth orbit (and probably won't be for a century at least). And there isn't anything worthwhile for humanity out there anyway. At least at the current stage in our "evolution". So for now manned spaceflight programmes are just vanity projects funded by politicians (for "national pride" or whatever) or some billionaire celebrities like Musk.

Also I don't think that world peace would be necessary for space colonization. It could be born out of conflict or for economic reasons, like colonization of Americas. It's simply that it will take centuries for us to reach a point when the prospect of leaving Earth will become attractive for regular people (if we survive that much of course).