It's not a negligible difference. The race results for that specific race are correlated with temperature during the race. The horses naturally hold back on their speed when it's hot out, to prevent running to exhaustion. This is borne out by statistical results comparing speeds for horse races generally, not just the human versus horse race.

Note that horses generally are much faster than humans, even over distance, so the trend lines wouldn't cross unless very hot, to where it would literally be dangerous for either species to be running.