I think Jefferson is an underrated contender here. The Louisiana Purchase set the stage for a lot of the subsequent acts of expansion/colonization/genocide. Maybe it's a naive view of history; some degree of expansion is probably inevitable, but it's possible that American expansion could have been slowed/reduced if they had more resistance from other European powers near the start (i.e. if they had to fight France to expand westward instead of getting carte blanche from them). There was not-insignificant opposition to the purchase and Jefferson had a role in pushing in favor of it, so perhaps you could argue that Jefferson has a significant responsibility in driving American colonialism and making it so powerful today. I think you could make an argument that he had a particularly harmful role in how he shaped early US history based on that (and of course due to the slave-owning and protecting slavery that most other early presidents did as well).