But it does affect the downward force acting on the object. Given two objects of the same shape but with different masses, one will indeed fall slower than the other. This is because the ratio of weight to surface area differs a lot between the two. Here's a calculator from NASA you can play with, and a relevant passage from the same page:

If we have two objects with the same area and drag coefficient, like two identically sized spheres, the lighter object falls slower. This seems to contradict the findings of Galileo that all free-falling objects fall at the same rate with equal air resistance. But Galileo’s principle only applies in a vacuum, where there is NO air resistance and drag is equal to zero.

https://www1.grc.nasa.gov/beginners-guide-to-aeronautics/termvel/