One of the first things they taught in my micro bio class is that no single bacteria is bad. What is bad is when they are where they shouldn't be. This makes the point of hand washing more or less to remove bacteria that may be problematic being on your hands where you could spread them to vulnerable areas.

Regular hand soap is perfectly fine. It may dry your hands out from overuse, but as soon as you touch any part of your body (like your arms or face) your hands would be inoculated with your body's natural flora that resides on your skin.

Antibacterial soap would be the larger issue, as a lot of them have what is called a persistent effect. This persistent effect means that the antibacterial agent leaves traces of itself behind and will kill any bacteria that is susceptible to it. While good if used responsibly for people like surgeons to eliminate/reduce any stray bacteria during a procedure, this can lead to the same sort of scenario we have with antibiotics where some bacteria may develop a resistance to it, which could also include pathogenic varients/strains.