You will pay tax on the growth, but not on the initial deposit. But that's just because you would be taxed twice.

That's a very good point! Compared to using an investment account, using a Roth IRA to save for a down payment can potentially save you 15ish% in capital gains taxes. Much like the 401k vs. IRA distinction, however, there are tradeoffs:

• ~~Roth IRAs and traditional IRAs both share from the same $6,500 annual contribution limit "pool". This effectively means that you're sacrificing the opportunity to use a traditional IRA for retirement savings whilst using a Roth IRA to save for a down-payment. If you're already saving under a 401k plan, this is basically a non-issue... but it's actually a complete showstopper if an IRA is your only available retirement savings option.~~ This bullet is not actually true, as pointed out elsewhere by @droans@lemmy.world. For clarity: contributions to a Roth IRA do not affect the limits associated with traditional IRAs and vice versa. You can contribute to a traditional IRA and that won't limit your ability to use this strategy.
• The main upside of using a Roth IRA to save for a down-payment is that you get to avoid paying capital gains taxes on profits generated by the IRA. This is because, unlike investment accounts, any gains produced from Roth retirement accounts are tax-exempt. It's effectively a ~15% discount on your payment![^1]
• The catch is that Roth IRAs are retirement accounts. You're not normally allowed to take the gains out early like this. The only reason the strategy works at all is because there's a special exception for this. In order to qualify, you must be a first-time home buyer and you must have had the Roth IRA account open for at minimum 5 years. You only get one chance to do this and you need to start laying the groundwork at least 5 years in advance!
• Another catch is that -- if you get cold feet about buying a home -- you basically lock yourself out of using that Roth growth money until you retire (unless you pay a hefty fee).[^2]

tl;dr: If you're a first-time home buyer ~~and you have spare IRA headroom that you weren't otherwise planning on using~~ and you have a healthy enough emergency fund that you can handle the risk(?) of accidentally saving more than intended for retirement, then there's a Roth IRA out there waiting with your name on it.

[^1]: This assumes that your gross yearly income is already > $40,000. If for whatever reason the sum total of your income is less than $40,000 during the year that you plan to buy your home (including the gains from the fund that you're about to cash in!), then you'll actually already be exempt from paying any capital gains tax. For non-retirees this is a pretty rare situation (unless you've been hiding something from the IRS...), but it prooobably happens? If this situation applies to you (and you don't mind getting audited), then you're basically free and clear to load up an investment account with as much money as you please and still reap those tax-free returns when the time comes -- no IRA shenanigans necessary.

[^2]: This is actually a pretty interesting tradeoff, because it's an upside for some and a downside for others. Take a hypothetical person with a $5,000 budget to split between retirement and saving for a down payment, for example: If they need to back out of buying a home, it's actually perfect that their down payment savings are locked into their IRA -- they were already planning on locking that money in and they still get to reap nice tax benefits in retirement. On the other hand, if that same person was financially counting on escaping from a bad rent situation... now they're going to come up short of where they had previously expected to be in terms of cash-on-hand.