In Europe you learn to respect walls at a very young age.
You don't deliberately kick a table leg with your toes either, you just know with certainty it will only give you pain.

Drywalls have some cushioning to them, they first compress then flex.
Brick is completely solid, it hurts even at very low speeds when hit with bone. Just knocking on it is painful.

Go outside, pick a nice flat pavement stone, put two sheets of paper over it. Now use your knuckles and knock around on it for a bit, then see what your instincts tell you when you think about punching that.