I have a rule about this (not that I don't break it at times). I only refactor in an unrelated story if it doesn't delay deliverables and existing tests cover the code.

And you're generally right about tech that not being prioritized, but you should have a talk with your product manager/owner to strike a deal for some small percentage of your work to include tech debt. We were able to convince ours that it was otherwise affecting our velocity.