Error when creating a pool from a custom image with azure python sdk

Question

I'm trying to create a pool using a custom image I created from VM with azure python sdk. The location and resource group match.

Here's my code:

import azure.batch as batch
from azure.batch import BatchServiceClient
from azure.batch.batch_auth import SharedKeyCredentials
from azure.batch import models



account = 'mybatch'
key = 'Adgfdj1hhsdfqATc/K2fgxdfg/asYgKRP2pUdfglBce7mgmSBdfgdhC7f3Zdfgrcgkdgh/dfglA=='
batch_url = 'https://mybatch.westeurope.batch.azure.com'

creds = SharedKeyCredentials(account, key)
batch_client = BatchServiceClient(creds, base_url = batch_url)

pool_id_base = 'mypool'
idx = 1

pool_id = pool_id_base + str( idx )

while batch_client.pool.exists( pool_id ):
  idx += 1
  pool_id = pool_id_base + str( idx )

print( 'pool_id ' + pool_id )

sku_to_use =  'batch.node.ubuntu 18.04'

# 
# image_ref_to_use = models.ImageReference(
#     offer = 'UbuntuServer', 
#     publisher = 'Canonical',
#     sku = '18.04-LTS', 
#     version = 'latest'
#   )



image_ref_to_use = models.ImageReference(
    virtual_machine_image_id = '/subscriptions/1834572sd-34sd409a-sdfb-sc345csdfesourceGroups/resource-group-1/providers/Microsoft.Compute/images/my-image-1'
  )


vm_size = 'Standard_D3_v2' 

vmc = models.VirtualMachineConfiguration(
  image_reference = image_ref_to_use,
  node_agent_sku_id = sku_to_use
)

new_pool = models.PoolAddParameter(
  id = pool_id, 
  vm_size = vm_size, 
  virtual_machine_configuration = vmc,
  target_dedicated_nodes = 1
)

batch_client.pool.add(new_pool)

According to the docs I should be able to use either virtual_machine_image_id other provide marketplace image parameters. I can create a pool of standard marketplace images, but I get an error when I'm trying to use an id of my custom image.

Traceback (most recent call last):   File "create_pool.py", line 60, in <module>
    batch_client.pool.add(new_pool)   File "/root/miniconda/lib/python3.6/site-packages/azure/batch/operations/pool_operations.py", line 312, in add
    raise models.BatchErrorException(self._deserialize, response) azure.batch.models.batch_error.BatchErrorException: {'additional_properties': {}, 'lang': 'en-US', 'value': 'Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:0dfdf9c1-edad-4b72-8e8f-f8dbcfd0abbdf\nTime:2018-12-06T10:51:21.9417222Z'}

How can I resolve this issue?

UPDATE

I tried to use ServicePrincipalCredentials with the following:

CLIENT_ID: I created a new application in Defaut Directiry -> Add registration and got it's Application ID.

SECRET: A created a key for the new application and used its value.

TENANT_ID: az account show in the cloud shell.

RESOURCE: Used 'https://batch.core.windows.net/'.

Updated my code like this:

from azure.common.credentials import ServicePrincipalCredentials

creds = ServicePrincipalCredentials(
  client_id=CLIENT_ID,
  secret=SECRET,
  tenant=TENANT_ID,
  resource=RESOURCE
)

And I get another error:

Keyring cache token has failed: No recommended backend was available. Install the keyrings.alt package if you want to use the non-recommended backends. See README.rst for details.
Traceback (most recent call last):
  File "create_pool.py", line 41, in <module>
    while batch_client.pool.exists( pool_id ):
  File "/root/miniconda/lib/python3.6/site-packages/azure/batch/operations/pool_operations.py", line 624, in exists
    raise models.BatchErrorException(self._deserialize, response)
azure.batch.models.batch_error.BatchErrorException: Operation returned an invalid status code 'Server failed to authorize the request.'

Possible duplicate of Azure Batch Pool: How do I use a custom VM Image via Python? — fpark

Shashank Verma Shashank Verma · Accepted Answer · 2018-12-06T15:18:29

Try using Service Principal Credentials instead of the Shared Key Credentials

credentials = ServicePrincipalCredentials(
    client_id=CLIENT_ID,
    secret=SECRET,
    tenant=TENANT_ID,
    resource=RESOURCE
)

There seems to be an error with Shared Key Credentials.

Documentation Link: https://docs.microsoft.com/en-us/azure/batch/batch-aad-auth

Issue Link: https://github.com/Azure/azure-sdk-for-python/issues/1668

Note: Please remove your account details as it can be accessed by anyone. Replace the account name and key with ****.

Update If the Service Principal Credentials are not working, try using the User credentials and see if that works.

from azure.common.credentials import UserPassCredentials
import azure.batch.batch_service_client as batch

credentials = UserPassCredentials(
    azure_user,
    azure_pass
)
batch_client = batch.BatchServiceClient(credentials, base_url = batch_url)

Error when creating a pool from a custom image with azure python sdk

1 Answers