Unable to download packages to Azure Artifacts from Azure Pipelines

Question

Requesting your help with Azure Artifact connection from Azure Pipelines.

My Azure Pipeline is building a image from docker file and a 'requirements' file has list of packages to be pip installed. In the pipeline I authenticate to my Azure Artifacts feed using PipAuthenticate@1 task and the authentication is successfull and the URL is passed as an argument to the docker file.

However, I can see that the packages are getting installed from external links, but are not downloaded to my artifact feed.

The artifact feed 'testartifact' is currently empty and so it is correctly going to the external link to download the package. But I was expecting the package to be then saved in 'testartifact' feed so that next docker build, it takes the package directly from testartifact feed. Is my assumption correct? If so, could you help on if I am missing something in the code due to which the package is not getting saved to my artifact.

Here is the Azure Pipeline yaml file and the docker file. Also attached the log of package download.

Thanks for your time!

pool:
  vmImage: 'ubuntu-latest'

# Set variables
variables:
  imageversion: 1.0
  artifactFeed: testartifact

stages:
- stage: DevDeploy
  jobs:
  - job: DevBuildandPushImage
    steps:
    - bash: echo DevDeploy 
    - task: PipAuthenticate@1
      displayName: 'Pip Authenticate'
      inputs:
        artifactFeeds: $(artifactFeed)
        onlyAddExtraIndex: true

    - bash: echo "##vso[task.setvariable variable=artifactoryUrl;]$PIP_EXTRA_INDEX_URL"
    - bash: echo $PIP_EXTRA_INDEX_URL

    - task: Docker@2
      inputs:
        containerRegistry: 'testcontaineregistry'
        repository: 'testrepository'
        command: 'build'
        Dockerfile: '**/dockerfile'
        arguments: '--build-arg PIP_EXTRA_URL=$(PIP_EXTRA_INDEX_URL)'

Part of the dockerfile

ARG PIP_EXTRA_URL
ENV PIP_EXTRA_INDEX_URL=$PIP_EXTRA_URL
RUN echo 'PIP_EXTRA_INDEX_URL'$PIP_EXTRA_INDEX_URL

# Install Python Packages & Requirements
COPY requirements requirements
RUN pip3 install -r requirements  --extra-index-url $PIP_EXTRA_URL

part of the log

2020-07-16T17:39:05.0301632Z Step 8/28 : RUN echo 'PIP_EXTRA_INDEX_URL'$PIP_EXTRA_INDEX_URL
2020-07-16T17:39:05.4787725Z PIP_EXTRA_INDEX_URLhttps://build:***@XXXXXXX.pkgs.visualstudio.com/_packaging/testartifact/pypi/simple
2020-07-16T17:39:06.1264997Z Step 9/28 : COPY requirements requirements
2020-07-16T17:39:07.0309036Z Step 10/28 : RUN pip3 install -r requirements  --extra-index-url $PIP_EXTRA_URL
2020-07-16T17:39:08.3873873Z Collecting pypyodbc (from -r requirements (line 1))
2020-07-16T17:39:08.7139882Z   Downloading https://files.pythonhosted.org/packages/ea/48/bb5412846df5b8f97d42ac24ac36a6b77a802c2778e217adc0d3ec1ee7bf/pypyodbc-1.3.5.2.zip
2020-07-16T17:39:08.9900873Z Collecting pyodbc (from -r requirements (line 2))
2020-07-16T17:39:09.2421266Z   Downloading https://files.pythonhosted.org/packages/81/0d/bb08bb16c97765244791c73e49de9fd4c24bb3ef00313aed82e5640dee5d/pyodbc-4.0.30.tar.gz (266kB)
2020-07-16T17:39:09.4960835Z Collecting xlrd (from -r requirements (line 3))
2020-07-16T17:39:09.6500787Z   Downloading https://files.pythonhosted.org/packages/b0/16/63576a1a001752e34bf8ea62e367997530dc553b689356b9879339cf45a4/xlrd-1.2.0-py2.py3-none-any.whl (103kB)
2020-07-16T17:39:09.6782714Z Collecting pandas (from -r requirements (line 4))
2020-07-16T17:39:10.2506552Z   Downloading https://files.pythonhosted.org/packages/c0/95/cb9820560a2713384ef49060b0087dfa2591c6db6f240215c2bce1f4211c/pandas-1.0.5-cp36-cp36m-manylinux1_x86_64.whl (10.1MB)
2020-07-16T17:39:11.4371150Z Collecting datetime (from -r requirements (line 5))
2020-07-16T17:39:11.6083120Z   Downloading https://files.pythonhosted.org/packages/73/22/a5297f3a1f92468cc737f8ce7ba6e5f245fcfafeae810ba37bd1039ea01c/DateTime-4.3-py2.py3-none-any.whl (60kB)
2020-07-16T17:39:11.6289946Z Collecting azure-storage-blob (from -r requirements (line 6))

Kevin Lu-MSFT Kevin Lu-MSFT · Accepted Answer · 2020-07-17T08:13:41

From the task log, the Python packages are restored from external links. You need to make sure that the packages are installed from Feed upstream source . Then the package will exist in the feed after installation.

Here are the steps:

Step1: Add the Python Upstream source to feed.

Step2: Use PipAuthenticate task to get the $PIP_EXTRA_INDEX_URL

Step3: Use the $PIP_EXTRA_INDEX_URL to install package from feed.

pip install -r requirements.txt --index-url $PIP_EXTRA_INDEX_URL

Note: The step 2 and 3 are already existing in your yaml file. But the pip install script seems to have issue. You need to directly add the --index-url parameter.

Then the packages are installed from feed upstream source.

In this case, these packages will also exist in the feed.

Unable to download packages to Azure Artifacts from Azure Pipelines

1 Answers