Depends in where you live. For instance in Germany house numbers are associated with the building but in France house numbers are their own object. The Idea there being that one should map the location where the plate with the houseumber is, and not the building(s) its associated with. If you dig throug osm map forums or issue trackers you'll find the dispute resurfacing every now and then.