reboot makes no difference. A new terminal gives the symptoms from the start.

I think I found a bad workaround. If I add this script to ~/.zshrc (because I'm not using bash but zsh)

SSH_AUTH_SOCK=/tmp/ssh-agent-$USER-socket
export SSH_AUTH_SOCK
if [ ! -S "$SSH_AUTH_SOCK" ]; then
    eval $(ssh-agent -a "$SSH_AUTH_SOCK")
fi

then it works. But I think I'm still using the ssh agent which I actually should not be using. At least it's asking for the passphrase every time, which is nice. Even in the same terminal after ssh logout.

EDIT: The first two lines do the trick as well:

SSH_AUTH_SOCK=/tmp/ssh-agent-$USER-socket
export SSH_AUTH_SOCK

EDIT: If I change this SSH_AUTH_SOCK to ANYTHING else, it also works. So /run/user/1000/gcr/ssh does not work. I gave ample permission to this file, so that cannot be the problem. Perhaps BECAUSE this is a file. I think the SSH_AUTH_SOCK should point to a nonexisting file because then it makes temporarily a special file that it needs. Ok I'm just shooting in the dark.