In my experience with my Apple Watch you have to activate the wallet functionality in order to pay for something by clicking the side button twice, which should make it harder for somebody to just walk around with a terminal charging random people. Phones usually need to be unlocked to make payments too. In theory NFC credit cards could be scanned like this, and if you're worried about that you can look into NFC blocking wallets... I'm not super worried about it, though, because usually you wouldn't be on the hook for such a fraudulent charge.