The whole point of ssh-agent is to remember your passphrase.

replace passphrase with private key and you're very correct.

passphrases used to login to servers using PasswordAuthentication are not stored in the agent. i might be wrong with technical details on how the private key is actually stored in RAM by the agent, but in the context of ssh passphrases that could be directly used for login to servers, saying the agent stores passphrases is at least a bit misleading.

what you want is:

• use Key authentication, not passwords
• disable passwordauthentication on the server when you have setup and secured (some sort of backup) ssh access with keys instead of passwords.
• if you always want to provide a short password for login, then don't use an agent, i.e. unset that environment variable and check ssh_config
• give your private key a password that fits your needs (average time it shoulf take attackers to guess that password vs your time you need overall to exchange the pubkey on all your servers)
• change the privatekey every time immediately after someone might have had access to the password protected privkey file
• do not give others access to your account on your pc to not have to change your private key too often.

also an idea:

• use a token that stores the private key AND is PIN protected as in it would lock itself upon a few tries with a wrong pin. this way the "password" needed to enter for logins can be minimal while at the same time protecting the private key from beeing copied. but even then one should not let others have access to the same machine (of course not as root) or account (as user, but better not at all) as an unlocked token could also possibly be used to place a second attacker provided key on the server you wanted to protect.

all depends on the level of security you want to achieve. additional TOTP could improve security too (but beware that some authenticator providers might have "sharing" features which could compromise the TOTP token even before its first use.