They can request and may be assigned an individual tax ID number (ITIN). Having an ITIN may allow them to open a bank account. It also makes it so they pay taxes which a surprising number want to do. Since they want to stay in the country, having a long history of paying taxes is a way to show you don't want to cause trouble and you just want to work. Hell, a lot of them probably know they got Al Capone on tax evasion and don't want to end up like him. Here's a report I just Googled, and possibly the source of the number which is an estimate.

On the other side is how much tax funding is used on them. Here's a report from the House of Representatives which states they use more in services (Medicare, WIC, sending kids to school, etc) than they pay. I can't speak to how accurate the report is considering the makeup and views publicly expressed by members of the committee. I'm personally leaning towards 'they don't, and can't, use as more than they contribute' because they're not eligible for many benefits.