It's not actually the amount of time that it takes that's the problem. With pistols that have safeties, the proper training is (usually) to turn the safety off when raising the gun. The problem is that it's a critical step you can mess up or forget to do under stress. Then you're left with a dead trigger having just pulled a gun in a situation you viewed as dangerous enough to require shooting someone. You're also stressed to hell and unlikely to think "oh yes, my safety!" Throw in that these kinds of situations are ones where half a second can make a big difference, and the saftey is just another thing that can go wrong.

There's certainly tradeoffs, since not having a safety means it's more likely your mistakes will result in a round being fired, but you can layer other procedures and devices to minimize that risk. In the end, it's a feature that even the gun community can't agree on, which is why some guns have them and some don't.