I do this a lot too, and I also experienced people becoming frustrated with me. I didn't enjoy that, so I now try to do this less. You may not have the same inclination as I do, but I will share something that I found helpful, in case you do.

The way I thought about it is that when you write an essay, or give a presentation, the structure is usually "Intro: here's what I'm doing to tell you; Main: I'm telling you the stuff; Conclusion: Here's what I just told you". It can feel repetitive, but a good intro primes people for the info they're about to receive, and a good conclusion puts a cap on things to make it easier to reflect on the whole.

Conveying the context before giving the yes/no answers makes it harder for people to make sense of the wider whole, because the meaning of statements is context dependent. For example, if the question at hand was "should we go to the shop?' and I stated "the shop is going to close soon", the actual meaning of that could be "we should go right now so we make it in time" or "We're not going to make it in time, we should go tomorrow". If I haven't started by reasoning by making it clear whether I'm "yes, let's go" Vs "no, let's wait", then people listening to my reasons have to hold both of these versions in their head, and puzzle out which is the most plausible take by fitting together fragments of interpretations. It can feel like information overload, and I found that people were more able to listen to context and reasonings if I led with a clear "thesis statement"