This is the same concept as a convection oven.

Cold, but still air? You're going to form a little heat bubble around you that makes it less bad. But as soon as a moderate breeze comes through, you lose that heat bubble. If you're wearing wind resistant clothes, you still have the layer of warm air at your skin.