I have a bunch of strings and pfx certificates, which I want to store in Azure Key vault, where only allowed users/apps will be able to get them. It is not hard to do store
The original question asked how to retrieve the stored PFX as an X509Certificate2 object. Using a Base64 process similar to that posted by Sumedh Barde above (which has the advantage of stripping the password), the following code will return a X509 object. In a real application, the KeyVaultClient should be cached if you're retrieving multiple secrets, and the individual secrets should also be cached.
public static async Task GetSecretCertificateAsync(string secretName)
{
string baseUri = @"https://xxxxxxxx.vault.azure.net/secrets/";
var provider = new AzureServiceTokenProvider();
var client = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(provider.KeyVaultTokenCallback));
var secretBundle = await client.GetSecretAsync($"{baseUri}{secretName}").ConfigureAwait(false);
string pfx = secretBundle.Value;
var bytes = Convert.FromBase64String(pfx);
var coll = new X509Certificate2Collection();
coll.Import(bytes, "certificatePassword", X509KeyStorageFlags.Exportable);
return coll[0];
}