Talk to a lawyer who specializes in employment and labor. Many firms will offer a free consult where you can speak with somebody, explain your situation, and they can then tell you whether or not it'll be worth pursuing further. If they tell you you have a case, then congrats, get ready for court.

If they tell you that you don't have a case, then most likely you should drop it. You could always try asking another firm for their opinion, as well, but if the first firm turned down a case with you, then there's probably a good reason for it. It sucks, but sometimes that's the reality of it.

You also mentioned not being able to afford the time off work to handle a legal case, which is also a really big concern that you should consider. Sometimes the most unjust part of justice is how much it costs to get it. While standing up for your morals and ensuring that those who have done wrong are punished for it is great, just make sure that you don't take yourself down in process.