Workspace Authentication: More than one token matches the criteria

3
votes

I constantly run into problems when working on Azure Compute Instances and trying to connect from the Jupyter Lab to the workspace.

With InteractiveLoginAuthentication I get the following message:

AuthenticationException: AuthenticationException:
    Message: Could not retrieve user token. Please run 'az login'
    InnerException More than one token matches the criteria. The result is ambiguous.
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "inner_error": {
            "code": "Authentication"
        },
        "message": "Could not retrieve user token. Please run 'az login'"
    }
}

With a Service Principal this one (SP is owner in the ML Workspace):

WorkspaceException: WorkspaceException:
    Message: No workspaces found with name=xxx in all the subscriptions that you have access to.
    InnerException None
    ErrorResponse 
{
    "error": {
        "message": "No workspaces found with name=xxx in all the subscriptions that you have access to."
    }
}

I had another workspace in a different subscription where I could resolve it by giving the tennant as an extra input to the InteractiveLoginAuthentication. This time, no chance.

The funny thing is, though, that I can login to the workspace via InteractiveLoginAuthentication when doing it from my local computer.

I supsected that some old tokens are cached somewhere so I tried to use the "Private browsing" function of my browser. Furthermore, I deleted /home/azureuser/.azure/accessTokens.json but no effect.

Maybe some of you had this problem before and have an idea?

For reference some sites I checked:

Update

When I run this code:

from azureml.core.authentication import InteractiveLoginAuthentication
interactive_auth = InteractiveLoginAuthentication(tenant_id='xxx')

ws = Workspace.get(name='xxx',
                   subscription_id='xxx',
                   resource_group='xxx',
                   auth=interactive_auth)

I get the following trace:

---------------------------------------------------------------------------
AdalError                                 Traceback (most recent call last)
/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_with_refresh(profile_object, cloud_type, account_object, config_object, session_object, config_directory, force_reload, resource)
   1820         auth, _, _ = profile_object.get_login_credentials(resource)
-> 1821         access_token = auth._token_retriever()[1]
   1822         if (_get_exp_time(access_token) - time.time()) < _TOKEN_REFRESH_THRESHOLD_SEC:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/_vendor/azure_cli_core/_profile.py in _retrieve_token()
    525                     return self._creds_cache.retrieve_token_for_user(username_or_sp_id,
--> 526                                                                      account[_TENANT_ID], resource)
    527                 use_cert_sn_issuer = account[_USER_ENTITY].get(_SERVICE_PRINCIPAL_CERT_SN_ISSUER_AUTH)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/_vendor/azure_cli_core/_profile.py in retrieve_token_for_user(self, username, tenant, resource)
    889         context = self._auth_ctx_factory(self._cloud_type, tenant, cache=self.adal_token_cache)
--> 890         token_entry = context.acquire_token(resource, username, _CLIENT_ID)
    891         if not token_entry:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in acquire_token(self, resource, user_id, client_id)
    144 
--> 145         return self._acquire_token(token_func)
    146 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in _acquire_token(self, token_func, correlation_id)
    127         self.authority.validate(self._call_context)
--> 128         return token_func(self)
    129 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in token_func(self)
    142             token_request = TokenRequest(self._call_context, self, client_id, resource)
--> 143             return token_request.get_token_from_cache_with_refresh(user_id)
    144 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/token_request.py in get_token_from_cache_with_refresh(self, user_id)
    346         self._user_id = user_id
--> 347         return self._find_token_from_cache()
    348 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/token_request.py in _find_token_from_cache(self)
    126         cache_query = self._create_cache_query()
--> 127         return self._cache_driver.find(cache_query)
    128 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/cache_driver.py in find(self, query)
    195                         {"query": log.scrub_pii(query)})
--> 196         entry, is_resource_tenant_specific = self._load_single_entry_from_cache(query)
    197         if entry:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/cache_driver.py in _load_single_entry_from_cache(self, query)
    123             else:
--> 124                 raise AdalError('More than one token matches the criteria. The result is ambiguous.')
    125 

AdalError: More than one token matches the criteria. The result is ambiguous.

During handling of the above exception, another exception occurred:

AuthenticationException                   Traceback (most recent call last)
/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in wrapper(self, *args, **kwargs)
    288                     module_logger.debug("{} acquired lock in {} s.".format(type(self).__name__, duration))
--> 289                 return test_function(self, *args, **kwargs)
    290             except Exception as e:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token(self)
    474         else:
--> 475             return self._get_arm_token_using_interactive_auth()
    476 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_using_interactive_auth(self, force_reload, resource)
    589         arm_token = _get_arm_token_with_refresh(profile_object, cloud_type, ACCOUNT, CONFIG, SESSION,
--> 590                                                 get_config_dir(), force_reload=force_reload, resource=resource)
    591         # If a user has specified a tenant id then we need to check if this token is for that tenant.

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in connection_aborted_wrapper(*args, **kwargs)
    325                 try:
--> 326                     return function(*args, **kwargs)
    327                 except AuthenticationException as e:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_with_refresh(profile_object, cloud_type, account_object, config_object, session_object, config_directory, force_reload, resource)
   1829             raise AuthenticationException("Could not retrieve user token. Please run 'az login'",
-> 1830                                           inner_exception=e)
   1831 

AuthenticationException: AuthenticationException:
    Message: Could not retrieve user token. Please run 'az login'
    InnerException More than one token matches the criteria. The result is ambiguous.
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "inner_error": {
            "code": "Authentication"
        },
        "message": "Could not retrieve user token. Please run 'az login'"
    }
}

During handling of the above exception, another exception occurred:

AdalError                                 Traceback (most recent call last)
/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_with_refresh(profile_object, cloud_type, account_object, config_object, session_object, config_directory, force_reload, resource)
   1820         auth, _, _ = profile_object.get_login_credentials(resource)
-> 1821         access_token = auth._token_retriever()[1]
   1822         if (_get_exp_time(access_token) - time.time()) < _TOKEN_REFRESH_THRESHOLD_SEC:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/_vendor/azure_cli_core/_profile.py in _retrieve_token()
    525                     return self._creds_cache.retrieve_token_for_user(username_or_sp_id,
--> 526                                                                      account[_TENANT_ID], resource)
    527                 use_cert_sn_issuer = account[_USER_ENTITY].get(_SERVICE_PRINCIPAL_CERT_SN_ISSUER_AUTH)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/_vendor/azure_cli_core/_profile.py in retrieve_token_for_user(self, username, tenant, resource)
    889         context = self._auth_ctx_factory(self._cloud_type, tenant, cache=self.adal_token_cache)
--> 890         token_entry = context.acquire_token(resource, username, _CLIENT_ID)
    891         if not token_entry:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in acquire_token(self, resource, user_id, client_id)
    144 
--> 145         return self._acquire_token(token_func)
    146 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in _acquire_token(self, token_func, correlation_id)
    127         self.authority.validate(self._call_context)
--> 128         return token_func(self)
    129 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in token_func(self)
    142             token_request = TokenRequest(self._call_context, self, client_id, resource)
--> 143             return token_request.get_token_from_cache_with_refresh(user_id)
    144 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/token_request.py in get_token_from_cache_with_refresh(self, user_id)
    346         self._user_id = user_id
--> 347         return self._find_token_from_cache()
    348 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/token_request.py in _find_token_from_cache(self)
    126         cache_query = self._create_cache_query()
--> 127         return self._cache_driver.find(cache_query)
    128 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/cache_driver.py in find(self, query)
    195                         {"query": log.scrub_pii(query)})
--> 196         entry, is_resource_tenant_specific = self._load_single_entry_from_cache(query)
    197         if entry:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/cache_driver.py in _load_single_entry_from_cache(self, query)
    123             else:
--> 124                 raise AdalError('More than one token matches the criteria. The result is ambiguous.')
    125 

AdalError: More than one token matches the criteria. The result is ambiguous.

During handling of the above exception, another exception occurred:

AuthenticationException                   Traceback (most recent call last)
<ipython-input-2-fd1276999d15> in <module>
      5                    subscription_id='00c983e5-d766-480b-be75-abf95d1a46c3',
      6                    resource_group='BusinessIntelligence',
----> 7                    auth=interactive_auth)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/workspace.py in get(name, auth, subscription_id, resource_group)
    547 
    548         result_dict = Workspace.list(
--> 549             subscription_id, auth=auth, resource_group=resource_group)
    550         result_dict = {k.lower(): v for k, v in result_dict.items()}
    551         name = name.lower()

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/workspace.py in list(subscription_id, auth, resource_group)
    637         elif subscription_id and resource_group:
    638             workspaces_list = Workspace._list_legacy(
--> 639                 auth, subscription_id=subscription_id, resource_group_name=resource_group)
    640 
    641             Workspace._process_autorest_workspace_list(

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/workspace.py in _list_legacy(auth, subscription_id, resource_group_name, ignore_error)
   1373                 return None
   1374             else:
-> 1375                 raise e
   1376 
   1377     @staticmethod

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/workspace.py in _list_legacy(auth, subscription_id, resource_group_name, ignore_error)
   1367             # azureml._base_sdk_common.workspace.models.workspace.Workspace
   1368             workspace_autorest_list = _commands.list_workspace(
-> 1369                 auth, subscription_id=subscription_id, resource_group_name=resource_group_name)
   1370             return workspace_autorest_list
   1371         except Exception as e:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/_project/_commands.py in list_workspace(auth, subscription_id, resource_group_name)
    386         if resource_group_name:
    387             list_object = WorkspacesOperations.list_by_resource_group(
--> 388                 auth._get_service_client(AzureMachineLearningWorkspaces, subscription_id).workspaces,
    389                 resource_group_name)
    390             workspace_list = list_object.value

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_service_client(self, client_class, subscription_id, subscription_bound, base_url)
    155         # in the multi-tenant case, which causes confusion.
    156         if subscription_id:
--> 157             all_subscription_list, tenant_id = self._get_all_subscription_ids()
    158             self._check_if_subscription_exists(subscription_id, all_subscription_list, tenant_id)
    159 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_all_subscription_ids(self)
    497         :rtype: list, str
    498         """
--> 499         arm_token = self._get_arm_token()
    500         return self._get_all_subscription_ids_internal(arm_token)
    501 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in wrapper(self, *args, **kwargs)
    293                     InteractiveLoginAuthentication(force=True, tenant_id=self._tenant_id)
    294                     # Try one more time
--> 295                     return test_function(self, *args, **kwargs)
    296                 else:
    297                     raise e

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token(self)
    473             return self._ambient_auth._get_arm_token()
    474         else:
--> 475             return self._get_arm_token_using_interactive_auth()
    476 
    477     @_login_on_failure_decorator(_interactive_auth_lock)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_using_interactive_auth(self, force_reload, resource)
    588         profile_object = Profile(async_persist=False, cloud_type=cloud_type)
    589         arm_token = _get_arm_token_with_refresh(profile_object, cloud_type, ACCOUNT, CONFIG, SESSION,
--> 590                                                 get_config_dir(), force_reload=force_reload, resource=resource)
    591         # If a user has specified a tenant id then we need to check if this token is for that tenant.
    592         if self._tenant_id and fetch_tenantid_from_aad_token(arm_token) != self._tenant_id:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in connection_aborted_wrapper(*args, **kwargs)
    324             while True:
    325                 try:
--> 326                     return function(*args, **kwargs)
    327                 except AuthenticationException as e:
    328                     if "Connection aborted." in str(e) and attempt <= retries:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_with_refresh(profile_object, cloud_type, account_object, config_object, session_object, config_directory, force_reload, resource)
   1828         if not token_about_to_expire:
   1829             raise AuthenticationException("Could not retrieve user token. Please run 'az login'",
-> 1830                                           inner_exception=e)
   1831 
   1832     try:

AuthenticationException: AuthenticationException:
    Message: Could not retrieve user token. Please run 'az login'
    InnerException More than one token matches the criteria. The result is ambiguous.
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "inner_error": {
            "code": "Authentication"
        },
        "message": "Could not retrieve user token. Please run 'az login'"
    }
}
  • azureml-sdk is on version 1.9.0
  • I can connect an authenticate from my local machine. Problems only occur when I want to work on a compute instance.
3
azureazure-machine-learning-serviceazureml

3 Answers

2
votes

Okay, here is the answer:

  • You work for company A which is on Azure.
  • You get access to company B's subscription.
  • Problem is: You are associated to A's AAD in ML-Studio.
  • You need to specify the tenant ID in the InteractiveLoginAuthentication like so:
interactive_auth = InteractiveLoginAuthentication(tenant_id=tenant_id)

workspace = Workspace.get(name=workspace_name,
                          subscription_id=subscription_id,
                          resource_group=resource_group,
                          auth=interactive_auth)
  • Now the important part: You need to use company B's tenant_id (I used company A's all the time since I thought that was my authentication point)
  • Of course, this is obvious while you read it...as it is to me now :)

Hope this helps you. Took me some time but learned a lot ;)

1
votes

I have met the same issue and following code that takes the tenant ID and uses interactive auth using AZURE ML SDK working fine.

import os

import azureml

from azureml.core import Workspace

from azureml.core.authentication import InteractiveLoginAuthentication

interactive_auth = InteractiveLoginAuthentication(tenant_id=" ")

ws = Workspace(subscription_id="",

               resource_group="",

               workspace_name="",

               auth=interactive_auth)

print("Found workspace {} at location {}".format(ws.name, ws.location))

These are the two other methods I would recommend you to try:

  1. Set the tenant ID from Azure shell and skip the ‘auth’ parameter to Workspace(…)

    az account set -s **********

  2. az account set -s ********** and the python SDK code would be

    from azureml.core.authentication import AzureCliAuthentication

    cli_auth = AzureCliAuthentication()

    pass cli_auth to auth parameter instead of a interactivelogin object

0
votes

This can usually happen due to two reasons:

  1. Your token is not for the correct tenant. In this case, you would need to pass the tenantId of the subscription which holds your workspace. The ServicePrincipalAuthentication class takes tenanatId as a parameter. Make sure you pass the correct value.

  2. The workspace in that subscription has a name with capital letters. This was a SDK side bug which was fixed around three months ago. Make sure you are on the latest SDK.

Can you share the SDK version you are using? Also, does the following error only happen from the compute instance, or does it happen even when you run the SDK from a different machine?

WorkspaceException: WorkspaceException: Message: No workspaces found with name=xxx in all the subscriptions that you have access to. InnerException None ErrorResponse { "error": { "message": "No workspaces found with name=xxx in all the subscriptions that you have access to." } }

A sample code snippet along with full stack trace may help us investigate this issue better.