2
You’re NTA for having a boundary and you’re NTA for being curt.
There are two ways to express displeasure. One is with words and the other is with tone. Tone is a perfectly valid way of doing it.
Your tone was curt, which was a signal that said “this isn’t okay”. And it’s not okay. We often forgive others for their transgressions, but him being late was a transgression and forgiving those does incur a cost.
I only think he would be hurt by your tone here if he were weak and oversensitive. You saying, in polite but short form, that you have places to be, is perfectly okay here.