Most situations requiring you to stealth aren’t going to have time for you to get out of your armor

That goes double for heavy armors like plate which takes upwards of twenty minutes to remove depending on the system you're using.