Accessing Azure KeyVault secrets only with a certificate installed on the machine

Question

I use the following code to access my Azure KeyVault

public static string GetKeyVaultSecret(string keyVaultName, string secretName)
{
    string secret = "";
    string secretUrl = $"https://{keyVaultName}.vault.azure.net/secrets/{secretName}";
    AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider();
    var keyVaultClient = new KeyVaultClient(
        new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
    Task.Run(async () => {
        var secretObject = await keyVaultClient.GetSecretAsync(secretUrl).ConfigureAwait(false);
        secret = secretObject.Value;
    }).GetAwaiter().GetResult();
    return secret;
}

This works perfectly when I am logged in under my account. But of a login as a service account I get the error:

Parameters: Connection String: [No connection string specified], Resource:
https://vault.azure.net, Authority: https://login.windows.net/5a47d63b-1b7e-4d2d-9333-750184dcbc99. 
Exception Message: Tried to get token using Active Directory Integrated Authentication.
 Access token could not be acquired. unknown_user_type: Unknown User Type

I would like only the certificate to be used to authenticate and authorize access to the KeyVault and not in addition any Azure Active Directory account