Azure ML studio - Container Registry Error while trying to submit a pipeline

Question

I'm having the following error while trying to submit an Azure ML Studio pipeline

Get credentials or pull docker image failed with err: error response from daemon: get https://lgcrmldev.azurecr.io/v2/azureml/azureml_977f5bda2f6f4f634482661c121c8959/manifests/latest: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.

The notebook python code I'm doing is something on these lines:

# create a Python script to do the actual work and save it in the pipeline folder:

%%writefile $experiment_folder/batch_online_retail.py
import os
import numpy as np
from azureml.core import Model
import joblib


# Called when the service is loaded
def init():
    global model
    
    # Load the model
    model_path = Model.get_model_path('Random_Forest_model')
    model = joblib.load(model_path)

def run(batch):
    try:
        result = []
        
    # Process each line
    for in range (len(batch)):
        # Read the comma-delimited data into an array
        data = np.genfromtxt(f, delimiter=',')        
        # Reshape into a 2-dimensional array for prediction (model expects multiple items)
        prediction = model.predict(data.reshape(1, -1))        
        # Append prediction to results
        resultList.append("{}: {}".format(os.path.basename(f), prediction[0]))
    return resultList

# Creating the run context
from azureml.core import Environment
from azureml.core.runconfig import DEFAULT_CPU_IMAGE
from azureml.core.runconfig import CondaDependencies

# Add dependencies required by the model
# For scikit-learn models, you need scikit-learn
# For parallel pipeline steps, you need azureml-core and azureml-dataprep[fuse]
cd = CondaDependencies.create(conda_packages=['scikit-learn','pip'],
                              pip_packages=['azureml-defaults','azureml-core','azureml-dataprep[fuse,pandas]'])

batch_env = Environment(name='batch_environment')
batch_env.python.conda_dependencies = cd
batch_env.docker.enabled = True
batch_env.docker.base_image = DEFAULT_CPU_IMAGE
print('Configuration ready.')

# Creating the ParallelRunStep
from azureml.pipeline.steps import ParallelRunConfig, ParallelRunStep
from azureml.pipeline.core import PipelineData

default_ds = ws.get_default_datastore()

output_dir = PipelineData(name='inferences', 
                          datastore=default_ds, 
                          output_path_on_compute='online-retail/results')

parallel_run_config = ParallelRunConfig(
    source_directory=experiment_folder,
    entry_script="batch_online_retail.py",
    mini_batch_size="5",
    error_threshold=10,
    output_action="append_row",
    environment=batch_env,
    compute_target=inference_cluster,
    node_count=2)

parallelrun_step = ParallelRunStep(
    name='batch-score-retail',
    parallel_run_config=parallel_run_config,
    inputs=[batch_data_set.as_named_input('online_retail_batch')],
    output=output_dir,
    arguments=[],
    allow_reuse=True
)

print('Steps defined')

and finally,

# Create an Azure ML experiment in your workspace, put the step into a pipeline and run it
from azureml.core import Experiment
from azureml.pipeline.core import Pipeline

pipeline = Pipeline(workspace=ws, steps=[parallelrun_step])
pipeline_run = Experiment(ws, 'online-retail-deployment-cf').submit(pipeline)
pipeline_run.wait_for_completion(show_output=True)

It's in this final step that I keep getting the error above.

My Container Registry has my user and Azure ML resource as a Contributor in the access control panel, so I don't think it's lack of permissions.

I've found this Microsoft page that seems to have a fix for the error I'm having: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-faq#docker-push-succeeds-but-docker-pull-fails-with-error-unauthorized-authentication-required

But I don't understand how can I implement the suggested fix. This is because the Docker image the notebook uses is inside the Compute Instance created in Azure ML which we have limited access.

Any ideas on what is the problem and how to fix it?

Thank you in advance, Carla

Charles Xu Charles Xu · Accepted Answer · 2021-02-19T09:12:25

According to the example here, I think you need to configure the environment variables for the docker images stored in the Azure Container Registry:

batch_env = Environment(name='batch_environment')
batch_env.python.conda_dependencies = cd
batch_env.docker.enabled = True
# Set the container registry information.
batch_env.docker.base_image_registry.address = "myregistry.azurecr.io"
batch_env.docker.base_image_registry.username = "username"
batch_env.docker.base_image_registry.password = "password"
batch_env.docker.base_image = "myregistry.azurecr.io/DEFAULT_CPU_IMAGE"

Azure ML studio - Container Registry Error while trying to submit a pipeline

1 Answers