As someone who used to be in a (casual) orchestra, I can tell you that the musicians can interpret the conductor because they've rehearsed it extensively beforehand. The conductor is really just there is remind the musicians to do the things that they've practiced beforehand.

As for the baton's movements, that's meant to indicate the speed that the music is played at. Nobody can keep perfect rhythm, and in a large orchestra, the echoes and travel speed of sound becomes especially disorienting. It will start to sound like you are playing off-time from the rest of the orchestra. In those cases, everyone has to ignore the sound of their music and only use the conductor to figure out where in the song they are, and they just have to trust that it'll sound correct to the audience