If that is your threat model you can put your phone number privacy to no one. Then I can't see you use Signal even if I have your contact with your phone number saved and adding you with your username won't show you as my phone contact with the same phone number.