Yeah, same as wood, who would build houses out of that? /s

Hay is great insulation and it's sealed with protective material anyway in modern houses. Also, even if not: Some tribes make houses out of hay or straw and those houses are quite great in their particular circumstances. There are experiments to use hay bales for structural support as well and it holds up surprisingly well.

Also, fun fact: wooden houses are less dangerous in case of fire than houses made of steel and concrete because the steel rod reinforcements start becoming soft at low temperatures (~200°C) and crash. Wooden houses announce crashing when burning, concrete buildings don't.