What's the difference between a docker and just installing an app on rented server space?

Functionally? Not much. But people who selfhost typically want everything on computers they own. Whether it's for learning purposes, or to not have their stuff on "someone else's computer" selfhosting usually means you're running software on computers you own, almost always within your own house.

Does each dock (?) have its own server? (Apache or nginx or whatever?)

Each docker image usually has a web server built in. The philosophy of docker is that it contains everything needed to run the app. Even a small linux OS (LibELEC or Alpine are favorites for docker images). So while you're not managing the web server in a docker image each docker image will have its own web server if web access is needed

Does each dock host a whole site, or do you have a dock for your database and a dock for your web app?

Docker the program is what runs all the docker images on a computer. Each docker image is built as per the software's developer. Some docker images will have a web app and a database combined into a single image, while others will expect a separate database server running independent of the image (It won't care if the database server is a docker image or not; just that it has access)