There are a few different physical systems that people are trying to build quantum computers with. Superconducting loops are one of the most promising ones, because of a halfway decent decoherence rate. And yeah, superconducts needing near 0K temperature to operate is a problem. It's just hard to scale up while everything needs to be so cold. Room-temp superconductivity would be a huge advantage.

But even then, the decoherence rates are still too high for any long quantum computation. Last I heard, the best qubits are maybe barely getting to good enough errors rates that quantum error correction would be possible - which is great, but 'possible' and 'practical' still have a significant gap between them.

So in short, basically everything about the hardware needs to be better; and its just very very hard. Probably too hard to ever achieve the dream of having arbitrary quantum computation. (But there is always the possibility of some big new idea that makes everything work better.)