Azure KeyVault - too many connections from Azure Functions

Question

We've got some Azure Functions defined in a class using [FunctionName] attributes from the WebJobs SDK. There are several functions in the class and they all need access to secrets stored in an Azure KeyVault. The problem is that we have many hundreds invocations of the functions a minute, and since each one is making a call to the KeyVault, KeyVault is failing with a message saying something like, "Too many connections. Usually only 10 connections are allowed."

@crandycodes (Chris Anderson) on Twitter suggested making the KeyVaultClient static. However, the constructor we're using for the KeyVaultClient requires a delegate function for the constructor, and you can't use a static method as a delegate. So how can we make the KeyVaultClient static? That should allow the functions to share the client, reducing the number of sockets.

Here's our KeyVaultHelper class:

public class KeyVaultHelper
{
    public string ClientId { get; protected set; }

    public string ClientSecret { get; protected set; }

    public string VaultUrl { get; protected set; }

    public KeyVaultHelper(string clientId, string secret, string vaultName = null)
    {
        ClientId = clientId;
        ClientSecret = secret;
        VaultUrl = vaultName == null ? null : $"https://{vaultName}.vault.azure.net/";
    }

    public async Task<string> GetSecretAsync(string key)
    {
        try
        {

            using (var client = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(GetAccessTokenAsync),
                new HttpClient()))
            {
                var secret = await client.GetSecretAsync(VaultUrl, key);
                return secret.Value;
            }
        }
        catch (Exception ex)
        {
            throw new ApplicationException($"Could not get value for secret {key}", ex);
        }
    }

    public async Task<string> GetAccessTokenAsync(string authority, string resource, string scope)
    {
        var authContext = new AuthenticationContext(authority, TokenCache.DefaultShared);
        var clientCred = new ClientCredential(ClientId, ClientSecret);
        var result = await authContext.AcquireTokenAsync(resource, clientCred);

        if (result == null)
        {
            throw new InvalidOperationException("Could not get token for vault");
        }

        return result.AccessToken;
    }
}

Here's how we reference the class from our functions:

public class ProcessorEntryPoint
{
    [FunctionName("MyFuncA")]
    public static async Task ProcessA(
        [QueueTrigger("queue-a", Connection = "queues")]ProcessMessage msg,
        TraceWriter log
        )
    {
        var keyVaultHelper = new KeyVaultHelper(CloudConfigurationManager.GetSetting("ClientId"), CloudConfigurationManager.GetSetting("ClientSecret"),
            CloudConfigurationManager.GetSetting("VaultName"));
        var secret = keyVaultHelper.GetSecretAsync("mysecretkey");
        // do a stuff
    }

    [FunctionName("MyFuncB")]
    public static async Task ProcessB(
        [QueueTrigger("queue-b", Connection = "queues")]ProcessMessage msg,
        TraceWriter log
        )
    {
        var keyVaultHelper = new KeyVaultHelper(CloudConfigurationManager.GetSetting("ClientId"), CloudConfigurationManager.GetSetting("ClientSecret"),
            CloudConfigurationManager.GetSetting("VaultName"));
        var secret = keyVaultHelper.GetSecretAsync("mysecretkey");
        // do b stuff
    }
}

We could make the KeyVaultHelper class static, but that in turn would need a static KeyVaultClient object to avoid creating a new connection on each function call - so how do we do that or is there another solution? We can't believe that functions that require KeyVault access are not scalable!?

Answer 1

You can use a memory cache and set the length of the caching to a certain time which is acceptable in your scenario. In the following case you have a sliding expiration, you can also use a absolute expiration, depending on when the secrets change.

public async Task<string> GetSecretAsync(string key)
{
    MemoryCache memoryCache = MemoryCache.Default;
    string mkey = VaultUrl + "_" +key;
    if (!memoryCache.Contains(mkey))
    {
      try
      {

          using (var client = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(GetAccessTokenAsync),
            new HttpClient()))
          {
               memoryCache.Add(mkey, await client.GetSecretAsync(VaultUrl, key), new CacheItemPolicy() { SlidingExpiration = TimeSpan.FromHours(1) });
          }
      }
      catch (Exception ex)
      {
          throw new ApplicationException($"Could not get value for secret {key}", ex);
      }
      return memoryCache[mkey] as string;
    }
}

Answer 2

try the following changes in the helper:

public class KeyVaultHelper
{
    public string ClientId { get; protected set; }

    public string ClientSecret { get; protected set; }

    public string VaultUrl { get; protected set; }

    KeyVaultClient client = null;

    public KeyVaultHelper(string clientId, string secret, string vaultName = null)
    {
        ClientId = clientId;
        ClientSecret = secret;
        VaultUrl = vaultName == null ? null : $"https://{vaultName}.vault.azure.net/";
        client = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(GetAccessTokenAsync), new HttpClient());
    }

    public async Task<string> GetSecretAsync(string key)
    {
        try
        {
            if (client == null)
                client = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(GetAccessTokenAsync), new HttpClient());

            var secret = await client.GetSecretAsync(VaultUrl, key);
            return secret.Value;
        }
        catch (Exception ex)
        {
            if (client != null)
            {
                client.Dispose();
                client = null;
            }
            throw new ApplicationException($"Could not get value for secret {key}", ex);
        }
    }

    public async Task<string> GetAccessTokenAsync(string authority, string resource, string scope)
    {
        var authContext = new AuthenticationContext(authority, TokenCache.DefaultShared);
        var clientCred = new ClientCredential(ClientId, ClientSecret);
        var result = await authContext.AcquireTokenAsync(resource, clientCred);

        if (result == null)
        {
            throw new InvalidOperationException("Could not get token for vault");
        }

        return result.AccessToken;
    }
}

now, the function can use a default static constructor to keep the client proxy:

public static class ProcessorEntryPoint
{
    static KeyVaultHelper keyVaultHelper;

    static ProcessorEntryPoint()
    {
        keyVaultHelper = new KeyVaultHelper(CloudConfigurationManager.GetSetting("ClientId"), CloudConfigurationManager.GetSetting("ClientSecret"), CloudConfigurationManager.GetSetting("VaultName"));
    }

    [FunctionName("MyFuncA")]
    public static async Task ProcessA([QueueTrigger("queue-a", Connection = "queues")]ProcessMessage msg, TraceWriter log )
    {           
        var secret = keyVaultHelper.GetSecretAsync("mysecretkey");
        // do a stuff

    }

    [FunctionName("MyFuncB")]
    public static async Task ProcessB([QueueTrigger("queue-b", Connection = "queues")]ProcessMessage msg, TraceWriter log )
    {
        var secret = keyVaultHelper.GetSecretAsync("mysecretkey");
        // do b stuff

    }
}

Answer 3

You don't actually want KeyVault to scale like that. It is protecting you from racking up unnecessary costs and slow behavior. All you need to do it save the secret for later use. I've created a static class for static instantiation.

public static class KeyVaultHelper
{
    private static Dictionary<string, string> Cache = new Dictionary<string, string>();

    public static async Task<string> GetSecretAsync(string secretIdentifier)
    {
        if (Cache.ContainsKey(secretIdentifier))
            return Cache[secretIdentifier];

        var kv = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(GetToken));
        var secretValue = (await kv.GetSecretAsync(secretIdentifier)).Value;
        Cache[secretIdentifier] = secretValue;
        return secretValue;
    }

    private static async Task<string> GetToken(string authority, string resource, string scope)
    {
        var clientId = ConfigurationManager.AppSettings["ClientID"];
        var clientSecret = ConfigurationManager.AppSettings["ClientSecret"];
        var clientCred = new ClientCredential(clientId, clientSecret);

        var authContext = new AuthenticationContext(authority);
        AuthenticationResult result = await authContext.AcquireTokenAsync(resource, clientCred);

        if (result == null)
            throw new InvalidOperationException("Failed to obtain the JWT token");

        return result.AccessToken;
    }
}

Now in your code, you can do something like this:

private static readonly string ConnectionString = KeyVaultHelper.GetSecretAsync(ConfigurationManager.AppSettings["SqlConnectionSecretUri"]).GetAwaiter().GetResult();

Now whenever you need your secret, it is immediately there.

NOTE: If Azure Functions ever shuts down the instance due to lack of use, the static goes away and is reloaded the next time the function is called. Or you can your own functionality to reload the statics.