Years and years of trying to be nice to people and them just using it against you, and treating you like shit is how you become comfortable being direct, and no nonsense.

Also remember not to worry too much about what that person is gonna do. In most cases even a manager cant even do all that much to you in retaliation, and if they do you have a case to take to HR. Don't just accept that you are a victim.

It seems to me that by speaking up when you're still calm, you can avoid erupting at an inopportune moment and causing the huge scene that might worry you.

I used to feel terrified about how people judged me--for good reason, based on how people treated me when I was young. Eventually, I grew utterly exhausted from trying to please everyone, after which it became much easier to speak up for myself.

Now I find it easy to offer a cheery "No, thanks" while acting like it's perfectly normal and leaving the other person to be confused and to deal with it.

I wish you peace as you work towards finding your voice.