Caching ssh passphrase with Magit?

I can't seem to get this working so i'm wondering how other people are doing this. I have ssh-agent running through a Guix home service, and in my .ssh/config i have this:

Host *
AddKeysToAgent yes

This works just fine if i perform a git action like committing or pushing in the terminal, the password stays cached and it doesn't prompt me again. In Emacs with Magit however, it keeps prompting me for the password every time, and running `ssh-add -l` shows that the ssh key does not get added to the agent. This is confusing to me cause Magit is just a front-end and still uses Git in the background right? Emacs also has `SSH_AUTH_SOCK` set correctly (it doesn't have the PID variable though, but my terminal doesn't seem to have that set either and yet it works just fine with Git in the terminal). I've looked at thing like ssh-agency, but i'm not sure if i should be using that, cause i'm not sure if that tries to start its own instance of ssh-agent, which i don't want since i'm already managing that with a Guix service. Since i don't want to be prompted on each login either since i don't always need to use my ssh keys, the only thing i can think of for now is to manually add the keys with `ssh-add` before i start doing my git actions in Magit, but there must be a better way to go about this right?

https://redd.it/1wxouel
@r_emacs