I started using Visual Studio Code, and I was trying to save my test project into GitHub, but Visual Studio Code is always asking for my GitHub credentials.
Maybe this is obvious to some (wasn't to me). It also doesn't solve the issue if you absolutely require a passphrase, but this was a decent compromise in my situation on Mac.