Yeah you're right. "Could be gone forever" is probably more accurate. I imagine most banks would give you a bit of leeway, just like most CCs aren't going to hold you liable for the first $50 like they technically could.
But at the same time, that money is gone immediately from your account, so if a sizable amount gets debited it could put you in a pickle until/if they refund it.
And...?