Are you puzzled about why ssh does not use your ssh-agent?

It could be that you are using the IdentityFile combined with the IdentitiesOnly option. When IdentitiesOnly is set to yes, ssh will not try to use the ssh agent, but only the key that you specified with IdentityFile. Consequently, you have to enter the passphrase every time.

Instead, use the CertificateFile option (and specify the corresponding public key). This way, ssh will use the ssh agent properly.

Please note that in that case, you will need to load the private key in the ssh-agent first, eg: $ ssh-add /PATH/TO/PRIVATE/KEY. If you don’t, ssh will not find the private key in the ssh agent, and has no real way to know what is the private key to use otherwise.