Yes, it can slow down downloading.

(The explanation below is simplified quite a bit)

When you download the server that is sending you the file doesn’t just dump all the data onto the network in one go. They don’t know how fast you can receive and it’s not like the routers along the way will buffer large amounts of data. It needs to figure out how fast it can send.

So how does it do this? The sender sends a few packets of data and then waits for the receiver to acknowledge reception before it sends more data. Now the acknowledgment message isn’t that big so when downloading the amount of data sent back (uploaded) is just a tiny fraction of the amount downloaded, so that usually doesn’t matter.

The problem occurs when your local network is much faster than your internet upload and your router isn’t smart about which packets to send first. A good router will not allocate all the spots in the outgoing queue to the connection doing the large upload and instead will make sure the connection with smaller amounts of outgoing data will get a fair turn.

If your router isn’t smart like that the ‘data received, please send more’ packets may be delayed because of all the other outgoing packets and thus slow down the download.