There some reasons:

• Deviation from the norm is always seen as dangerous for a lot of people. The average man usually is able and will adapt to change but some of them don't.
• Scarecrow tactics in politics. "Gay and trans people kill the christian family values", " Gay people mindwash our children" (reference: my country, aka Hungary)
• Others (For example a friend of a friend was at a party, met with a girl and she later on spilled the beans just before the sex act. She was trans and that guy gut furious. Idk him probably he was already a transphobe before)

EDIT: Sry didn't read the title