Accessing Azure Key Vault from Azure Batch

Question

I am trying to access secrets in my Azure Key Vault from VMs running in my Azure Batch node pool.

However, I keep running into the exception:

Exception Message: Tried 1 certificate(s). Access token could not be acquired.

Exception for cert #1 with thumbprint MY-THUMBPRINT: Keyset does not exist

So far I have been following the instructions outlined here: https://docs.microsoft.com/en-us/azure/key-vault/service-to-service-authentication

The article outlines the scenario of Azure Batch, and indicates I should use a Service Principal. I would like to ensure no secrets or keys are in version control, so I am using the first method of a certificate in a local keystore to sign into Azure AD.

Running all of the below locally as an executable works fine, but fails when run on an Azure Batch pool node.

My steps so far to do the above are:

1. Create a service principal and associated certificate in keyvault: az ad sp create-for-rbac --name myserviceprincipal --create-cert --cert mycertname --keyvault mykeyvaultname. Keep the service principal app id and tenant id for use in the AzureServicesAuthConnectionString.

2. Create key vault access policy for the created service principal (done in the azure portal UI).

3. Download the created certificate in PFX/PEM format (done in the Azure Portal UI).

4. Ensuring a PFX password on the certificate (I am doing this as uploading the cert to azure batch in step 6 requires an associated password): https://coombes.nz/blog/azure-keyvault-export-certificate/

# Replace these variables with your own values
$vaultName = "YOUR-KEYVAULT-NAME"
$certificateName = "YOUR-CERTIFICATE-NAME"
$pfxPath = [Environment]::GetFolderPath("Desktop") + "\$certificateName.pfx"
$password = "YOUR-CERTIFICATE-PASSWORD"

$pfxSecret = Get-AzureKeyVaultSecret -VaultName $vaultName -Name $certificateName
$pfxUnprotectedBytes = [Convert]::FromBase64String($pfxSecret.SecretValueText)
$pfx = New-Object Security.Cryptography.X509Certificates.X509Certificate2
$pfx.Import($pfxUnprotectedBytes, $null, [Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable)
$pfxProtectedBytes = $pfx.Export([Security.Cryptography.X509Certificates.X509ContentType]::Pkcs12, $password)
[IO.File]::WriteAllBytes($pfxPath, $pfxProtectedBytes)

5. Installing the certificate locally into my computer's LocalMachine store (for local testing).

6. Uploading the certificate into Azure Batch (using the Azure Portal UI upload).

7. Associating the certificate with the appropriate Node Pool and rebooting the nodes (using the Azure Portal UI for now).

My application package running on Azure Batch is a simple console executable. The AzureServicesAuthConnectionString is set to RunAs=App;AppId={AppId};TenantId={TenantId};CertificateThumbprint={Thumbprint};CertificateStoreLocation={LocalMachine} and the remaining key vault code to retreive the secret looks like:

Environment.SetEnvironmentVariable("AzureServicesAuthConnectionString", "RunAs=App;AppId=<MY-APP-ID>;TenantId=<MY-TENANT>;CertificateThumbprint=<MY-THUMBPRINT>;CertificateStoreLocation=LocalMachine");

AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider(Environment.GetEnvironmentVariable("AzureServicesAuthConnectionString"));
KeyVaultClient keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
var secret = await keyVaultClient.GetSecretAsync("<MY-SECRET>").ConfigureAwait(false);
var message = secret.Value;
Console.WriteLine(message);

Things work fine locally, but fail on the remote node. I am able to RDP into the Azure Batch node and see that the certificate has been installed for the local machine.

I am wondering how to resolve my error or if my above steps are wrong in some way?

Answer 1

In addition to the above, I have followed the instructions from Sam Cogan article here: https://samcogan.com/secure-credential-access-with-azure-batch-and-keyvault/

However I receive the same issue as the original question. My error and associated repro steps for the Sam Cogan article are:

Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException: Parameters: Connection String: RunAs=App;AppId=<My Service Principal ID>;TenantId=<My Tenant ID>;CertificateThumbprint=<My cert thumbprint>;CertificateStoreLocation=LocalMachine, Resource: https://vault.azure.net, Authority: https://login.windows.net/<My Tenant ID>. Exception Message: Tried 1 certificate(s). Access token could not be acquired.
Exception for cert #1 with thumbprint <My cert thumbprint>: Keyset does not exist

My repro steps are:

1. Create the certificate (I added the -a sha256 -len 2048 part as I had the same error outlined and resolved here: How to authenticate with Key Vault from Azure Batch)

C:\Program Files (x86)\Windows Kits\10\bin\x64> .\makecert.exe -sv batchcertificate6.pvk -n "cn=andybatch6.cert.mydomain.org" batchcertificate6.cer -r -pe -a sha256 -len 2048

2. Convert the certificate in step 1 to PFX format:

C:\Program Files (x86)\Windows Kits\10\bin\x64> .\pvk2pfx.exe -pvk batchcertificate6.pvk -spc batchcertificate6.cer -pfx batchcertificate7.pfx -po <MyPassword>  -pi <MyPassword>  

3. Create the ActiveDirectory Application and Service Principal and associate and upload the created cert:

#Point the script at the cer file you created 

$cerCertificateFilePath = 'C:\Program Files (x86)\Windows Kits\10\bin\x64\batchcertificate6.cer' 
$cer = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 
$cer.Import($cerCertificateFilePath)

#Load the certificate into memory 
$credValue = [System.Convert]::ToBase64String($cer.GetRawCertData()) 

#Create a new AAD application that uses this certifcate 
$newADApplication = New-AzureRmADApplication -DisplayName "<My display name>" -HomePage "<my url>" -IdentifierUris " <my url>" -certValue $credValue 

#Create new AAD service principle that uses this application 
$newAzureAdPrincipal = New-AzureRmADServicePrincipal -ApplicationId $newADApplication.ApplicationId

4. Grant Key Vault Access for the Service Principal (I added extra permissions in the UI but also ran this via command line):

Set-AzureRmKeyVaultAccessPolicy -VaultName 'myvaultname' -ServicePrincipalName '<my url>' -PermissionsToSecrets 'Get'

5. Upload the Cert (pfx) to Azure Batch via the portal. Associate it with my node pool as LocalMachine and reboot the nodes. Run the application with a connection string like:

Environment.SetEnvironmentVariable("AzureServicesAuthConnectionString",
                    "RunAs=App;" +
                    "AppId=<the app id of my active directory app registration> ;" +
                    "TenantId=<my subscription tenant id>;" +
                    "CertificateThumbprint=<the thumbprint of my cert>;" +
                    "CertificateStoreLocation=LocalMachine");

                AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider(Environment.GetEnvironmentVariable("AzureServicesAuthConnectionString"));
                KeyVaultClient keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
                var secret = await keyVaultClient.GetSecretAsync("https://<my vault name>.vault.azure.net/secrets/<secret name>/<secret id>")
                        .ConfigureAwait(false);
                message = secret.Value;
                Console.WriteLine(message);

My code works with the installed cert locally using the above connection string, but has the above mentioned exception when running on Azure Batch.

Answer 2

In order to access the Certificate, it must be associated and installed for the 'Current User.' It may be that LocalMachine does not have the appropriate level of permissions?

On Azure Batch, ensure that the uploaded certificate is associated with:

Store Name: 'My' Store Location: 'CurrentUser'

This post was helpful: https://github.com/nabhishek/customactivity_sample/tree/linkedservice

As was this post: X509Certificate - Keyset does not exist

The connection string in the C# exe looks like:

Environment.SetEnvironmentVariable("AzureServicesAuthConnectionString",
                    "RunAs=App;" +
                    "AppId=<the app id of my active directory app registration> ;" +
                    "TenantId=<my subscription tenant id>;" +
                    "CertificateThumbprint=<the thumbprint of my cert>;" +
                    "CertificateStoreLocation=CurrentUser");

                AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider(Environment.GetEnvironmentVariable("AzureServicesAuthConnectionString"));
                KeyVaultClient keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
                var secret = await keyVaultClient.GetSecretAsync("https://<my vault name>.vault.azure.net/secrets/<secret name>/<secret id>")
                        .ConfigureAwait(false);
                message = secret.Value;
                Console.WriteLine(message);