There are two concepts at play here: open-source and free software. An early example of open-source is AT&T Research UNIX, which was made source-available (for a few) to universities for research purposes, who could recompile the code and use the binaries for that purpose. Here, the use of the software is restricted by the license terms.

On the free software side, as a reimplementation if the Unix software utilities -- ie all the programs like tar, ps, sh -- GNU coreutils is GPL licensed, meaning any use of the compiled binaries is allowed, but there are restrictions on the distribution, of both source and binaries. As it turns out, GPL is both free and open-source (FOSS); there are fewer major examples of free but non-open source, but WinRAR and nVidia drivers on Linux would count.

Specifically, GPL and other copyleft licenses require that if you distribute the binary, you must make the source available under the same terms. If you've made no changes, then this is as simple as linking to the public source code repo. If you did add or remove code, you must release those alongside the binaries. If you simply use the binaries internally, you don't need to release anything at all, and can still use them for any internal purpose.

wouldn't GPL and other copyleft licenses be considered non-free as well since you are not free to do whatever you want with the source

From the background above, free software has always been understood to mean the freedom to use software, not necessarily distribute it. GPL complies with that definition for using the software, but also enforced a self-perpetuating distribution requirement. Unlike plain ol free software, under GPL, you must redistribute source if you distribute the software for use (aka binaries), and you must make that source also GPL.