Unable to pip install in Docker image as agent through Jenkins declarative pipeline

Question

I have yet another issue with permissions running Docker through Jenkins declarative pipeline. I want to build and publish a Python package through a Jenkins job in a Docker container:

pipeline {

  agent {
    docker {
      image 'python:3.7'
      label 'docker && linux'
    }
  }

  environment {
    PACKAGE_VERSION = readFile 'VERSION'
  }

  stages {

    stage('Package') {
      steps {
        sh 'python -V'
        sh 'python -m pip install -r requirements.txt --user --no-cache'
        sh 'python setup.py sdist'
      }
    }

    stage('Deploy') {
      steps {
        ...
      }
    }

  }

  post {
    always {
      cleanWs()
    }
  }

}

However, I am not allowed to pip install due to a PermissionError:

+python -m pip install -r requirements.txt --user --no-cache Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (40.0.0) Collecting pytest (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/9e/a1/8166a56ce9d89fdd9efcae5601e71758029d90e5644e0b7b6eda07e67c35/pytest-3.7.0-py2.py3-none-any.whl (202kB) Collecting py>=1.5.0 (from pytest->-r requirements.txt (line 2)) Downloading https://files.pythonhosted.org/packages/f3/bd/83369ff2dee18f22f27d16b78dd651e8939825af5f8b0b83c38729069962/py-1.5.4-py2.py3-none-any.whl (83kB) Collecting more-itertools>=4.0.0 (from pytest->-r requirements.txt (line 2)) Downloading https://files.pythonhosted.org/packages/79/b1/eace304ef66bd7d3d8b2f78cc374b73ca03bc53664d78151e9df3b3996cc/more_itertools-4.3.0-py3-none-any.whl (48kB) Collecting pluggy>=0.7 (from pytest->-r requirements.txt (line 2)) Downloading https://files.pythonhosted.org/packages/f5/f1/5a93c118663896d83f7bcbfb7f657ce1d0c0d617e6b4a443a53abcc658ca/pluggy-0.7.1-py2.py3-none-any.whl Collecting six>=1.10.0 (from pytest->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl Collecting atomicwrites>=1.0 (from pytest->-r requirements.txt (line 2)) Downloading https://files.pythonhosted.org/packages/0a/e8/cd6375e7a59664eeea9e1c77a766eeac0fc3083bb958c2b41ec46b95f29c/atomicwrites-1.1.5-py2.py3-none-any.whl Collecting attrs>=17.4.0 (from pytest->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/41/59/cedf87e91ed541be7957c501a92102f9cc6363c623a7666d69d51c78ac5b/attrs-18.1.0-py2.py3-none-any.whl Installing collected packages: py, six, more-itertools, pluggy, atomicwrites, attrs, pytest

Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/.local' Check the permissions.

How do I fix these permissions?

Answer 1

I have found what I myself would think is the prettier solution:

stage("Python Test") {
  agent { 
    docker {
      label "docker && linux" 
      image "python:3.7"
    }
  }
  steps {
    withEnv(["HOME=${env.WORKSPACE}"]) {
      sh "pip install -r requirements.txt --user"
      # python stuff
    }
  }
  post {
    cleanup {
      cleanWs()
    }
  }
}

This workaround steers completely around the issue itself, installing the packages at user level. The issue here was that the HOME-directory was not initially writeable either, thus overwriting the HOME directory.

Answer 2

I had a very similar pipeline that I was running right after setting up Docker agents on my Jenkins system, so I thought my setup was wrong. Using the comments in your thread, I cooked up this solution:

First, you'll need to be root inside your container, so change your agent declaration to be similar to this:

agent {
    docker {
        image "python:3.7"
        args '--user 0:0'
    }
}

Now I was able to use pip install! However, subsequent runs of the job would try to run git clean and fail since the built files inside the container were created by root. To fix that, I ran the clean command inside the container as my last step:

steps {
    sh 'git clean -fdx'
}

Update:

I found a problem where a failed build wouldn't clean up and killed all of the builds after it. To fix this, I put the clean action as a post-build task that always runs:

post {
    cleanup {
        cleanWs()
    }
}

Answer 3

withEnv(["HOME=${env.WORKSPACE}"]) {} is the key. Also, it can be used like:

node {
    def customImage = docker.build(...)
    customImage.inside {
        withEnv(["HOME=${env.WORKSPACE}"]) {
            sh '...'
        }
    }
}

Answer 4

You can try executing it as sudo:

 stage('Package') {
      steps {
        sh '''
            python -V
            sudo python -m pip install -r requirements.txt --user --no-cache
            sudo python setup.py sdist
           '''
      }
    }

You may have issues due Jenkins is not able to run command as sudo in that case I will recommend you to follow the steps mentioned in this article or in this SO question