The housing crisis arises out of one problem, and one problem only:

Housing as an investment.

My city has a rental vacancy rate of <4%, and a homeowner vacancy rate of <1%. Flippers leave a house empty while under the process of flipping it, and that's not what the numbers show. Landlords do increase the cost compared to ownership (they have to cover all normal costs of ownership, plus have profit for themselves), but they don't reduce the number of shelters being occupied. Not when vacancy rates are this low.

In other words, my particular city may have costs driven up by flippers and landlords, but the number of dwelling units would be short even without them. Getting rid of them would be an insufficient solution, even if there are some benefits on costs. It does not address the problem that we need more dwelling units.