Mainly because its more surfaces to touch.

I don't mind it so much on the way out because I'm washing my hands, but on the way in it's just feels icky because people are generally disgusting.