Playing Red Dead Redemption makes me think that at one point they weren't that expensive if you lived in a very rural area.

• Feeding them probably wasn't too expensive if you had a place they could just graze. Even if you didn't own a farm, there were probably still wild / common areas where animals could graze.
• Shoeing / vet care probably wasn't as expensive when horses were the main means of transportation, so vets and smiths were everywhere
• In a rural area, you probably already had a barn / stable / shack that you could use to provide the horse with shelter, so it didn't need its own additional building. If you did need to build a structure, land was cheap and so it was only the cost of labor you had to worry about.
• Cleaning out the horse poop was a chore, but it could be used as fertilizer, so it wasn't just something you had to dispose of
• You'd still need saddles, stirrups, reins, etc. But, that was made from leather and metal and would probably last decades with some basic maintenance
• Since horses were, ahem, workhorses, not race horses or display horses, they were probably bred to be sturdier and not as prone to requiring medicine or frequent vet trips

It was probably similar to cars today, where some people had expensive, fancy horses that they spent lots of money on, and other people had old clunkers that they got cheap and then rode until they died.

I get the impression that when people today talk about hoses being expensive, a lot of that expense is due to them living in a city. My guess is that if you already live on a working farm, adding one horse is not going to massively increase your expenses.