If you've got a great story idea, I'd write that first and then make the world match the story's needs. However, you want to make sure the world would function logically the same way even if the plot weren't carried out. That is to say, don't make absurdly convenient world building choices just to make the plot work. Readers will often pick up on that. "How convenient that the social tradition requires a man and a woman go on this adventure together so the main characters could fall in love!"

Sometimes the opposite happens and writers will spend so much effort on the world building that the story will suffer because the writer can't kill their darlings and cut out unnecessary details.

It's a balancing act.

Building a story is world building.

You just have to keep good notes. There's absolutely no meaningful difference in coming up with whatever system you're using and then telling stories with it, vs telling the story as you build the world, with the story guiding the building. You end up in the same place, you do the same amount of editing along the way, and no reader will be able to tell which you did.

I'm the exact opposite. I get lost in an ocean of detail while having issues coming up with a strong main thread.

The way different people write is usually split along the lines of this question. A good example demonstrating this is the Game of Thrones TV series. Martin's novels were very much written with the world building at the forefront; character's narratives were mostly secondary to what was going on in the world and if a character ended up in a deadly situation then plot armour wasn't going to save them (usually). However when the content of the novels ran out and the hollywood writers took over the plot, you almost immediately see a switch to character driven narratives because that's almost always how hollywood writes stories. Notice that basically no major characters die in the later seasons.

I think this difference is very noticeable in the TV series, even if people don't necessarily recognise this specific explanation of why the later seasons feel so different. And as a writer I think you want to at least recognise which of these styles you are personally using. As others have pointed out, if your worldbuilding seems too obviously constructed to make the plot work then that can be noticeable by the readers, but maybe that's fine if you're writing a very character focused narrative. You could even leave the details of the world fairly vague and ill defined if this is the case.

But if you want your world to feel detailed and cohesive, then I think you need to have a decent amount of the worldbuilding fleshed out before you start writing the characters. Like, by all means have your plot structure figured out, but the actual page to page writing should be informed by the worldbuilding otherwise there's a strong chance your world just feels like a cardboard backdrop to your characters.

Seems like it's kind of a both/and situation. "Why did the character do this? Hmm, because of space elves! Now where do they fit in to the world?"