I like the comparison to measuring physical attributes, but I think a better analogy for IQ testing would be "trying to measure athleticism by trying to measure who's best at playing basketball."

Defining "athleticism" itself is hard. It might include reaction time, coordination, strength, speed, flexibility, etc. There are some naturally occurring differences in those attributes, but to a degree they are also changeable. Any bottom-line, all-in measure is going to include some arbitrary decisions about the relative value of each attribute.

Trying to measure athleticism by who's best at basketball adds another layer of problems. Basketball (analogous to test taking, cultural context, etc.) involves skills that are separate from whatever you're calling athleticism. It's also a game where a big factor in success -- height -- is also probably something a lot of people would consider separate from athleticism.