It could be that you are using the
IdentityFile combined with the
IdentitiesOnly option. When
IdentitiesOnly is set to
yes, ssh will not try to use the ssh agent, but only the key that you specified with
IdentityFile. Consequently, you have to enter the passphrase every time.
Instead, use the
CertificateFile option (and specify the corresponding public key). This way, ssh will use the ssh agent properly.
Please note that in that case, you will need to load the private key in the ssh-agent first, eg:
$ ssh-add /PATH/TO/PRIVATE/KEY. If you don’t, ssh will not find the private key in the ssh agent, and has no real way to know what is the private key to use otherwise.