every day there's this underlying angst in my mind replaying over and over again, "I would not be attracted to the kind of person who might be attracted to me living this way."

I guess what you're looking for is someone who likes you enough not to care then rather than someone actively attracted to you living that way if that makes sense.

My wife was a single mum living in a pretty shitty apartment when we met initially. We're married 19 years now. I adopted that lovely boy and we have a heap more kids together. She's a beautiful soul and that's what I was looking for in life. Many people just look past things like your current living arrangement if you're right for them is what I'm saying I suppose.