Poor communities have worse public schools, fewer educational programs, etc. They have less access to education and thus have a harder time to excel in it. Affirmative action from my understanding was a way to offset the systemic racism favoring rich white communities. I don't think it's a good solution but removing it without a decade of solving the underlying issues and seeing the first kids with equal chances make it into university is just a horrible thought. EDIT: typo