what is the bare minimum of security measures you can do?

I guess just the normal things with p2p stuff: make sure no ports are exposed except for the essentials, update software, use SSL wherever possible.

When you don't use VPN, people will see your actual IP adress and will launch the same kind of attacks, they also launch on servers [1] to try to hijack your system and add them to their bot net.

[1] port scans, login-attemps, applying known exploits. If this doesn't sound scary, you should try operating a server that is exposed on the internet and then look at the number of login attemps.