Cannot access Azure Key Vault from Azure Data Lake Analytics

Question

I have an U-SQL script with custom extractor, which access Azure Key Vault to get some credentials.

I followed this tutorial. And I have equivalent code to get token from AD and then to call provided URI for actual credentials:

public static async Task<string> GetToken(string authority, string resource, string scope)
{
    var authContext = new AuthenticationContext(authority);
    var clientCred = new ClientCredential(applicationId, authenticationKey);
    AuthenticationResult result = await authContext.AcquireTokenAsync(resource, clientCred);
    if (result == null)
    {
        throw new InvalidOperationException("Failed to obtain the AD token");
    }
    return result.AccessToken;
}

public static async Task<string> GetSecret(string secretUri)
{
    var keyVaultClient = new KeyVaultClient(
            new KeyVaultClient.AuthenticationCallback(GetToken)
    );
    var sec = await keyVaultClient.GetSecretAsync(secretUri);
    return sec.Value;
}

My credentials were put into vault successfully, and I have an URI to access them - something like:

https://my-key-vault-name.vault.azure.net:443/secrets/MyCredentialsName/123abc

I've registered my app in Azure AD and got application-id and authentication-key for it and I allowed my app to read secret from Key Vault. In my U-SQL script I've referenced all needed assemblies.

When I run my script locally everything works great (that means connection from local machine to AD and to Key Vault are OK), but when I submit it for execution on remote Data Lake Analytics account I got the following error:

The remote name could not be resolved: 'my-key-vault-name.vault.azure.net'

at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)

My administrative rights on Azure resource group are limited, but I can access Firewall tab on Data Lake Analytics blade - I've tried enabling and disabling firewall, switching on/off Allow access to Azure services, still the error persists.

As dependencies, I am referencing Microsoft.Azure.KeyVault 2.0.6, Microsoft.Azure.KeyVaultWebKey 2.0.4, Microsoft.IdentityModel.Clients.ActiveDirectory 3.13.9.

Any ideas on how can I attempt to resolve it?

Michael Rys Michael Rys · Accepted Answer · 2017-05-02T02:33:44

U-SQL code running in ADLA does not allow you to connect to resources outside the container/VM. The reason is:

U-SQL's custom code calls possibly scaled out over 100 to 1000s of containers getting invoked millions for millions of rows. This can easily lead to a (hopefully unintended) distributed denial of service attach against the service you are trying to reach, leading to possibly DDOSing the service and getting the Azure IP ranges blocked.

Local run does not currently run in a container so has no such limit enforcement.

What are you trying to achieve with this call? Note that the data in storage can already be transparently encoded with Azure Key Vault.

Cannot access Azure Key Vault from Azure Data Lake Analytics

1 Answers