Fair point, but the difference is that human drivers are already at roughly their limit for how good they can drive, but self driving cars have the potential to exceed us.

It's similar to one of the biggest arguments in electric vs gas cars. Even if electric cars today are just as environmentally unfriendly as gas cars (they're not) the difference is that gas technology is super mature and there's very little improvements to be made by spending more money on it, electric battery technology on the other hand, is still in it's relative infancy and has huge potential to improve in numerous ways, but that can only happen if more people buy electric so more R&D money can be spent on it.