Run jenkins agent in docker container, issue with jenkins pipeline and /var/run/docker.socket

1
votes

I've created image that use for jenkins agent slave Here it is: https://hub.docker.com/r/kondaurov/jenkins_agent/~/dockerfile/

Than i run it:

docker run -d -p 2222:22 --name=jenkins_agent -v /var/run/docker.sock:/var/run/docker.sock -v kondaurov/jenkins_agent

Then i create new node in jenkins and it connects successfully via ssh

I try to run this pipeline:

pipeline {

    agent {
            docker {
                label 'agent01'
                image 'hello-world'
            }


    }

    stages {

        stage('Compile') {

            steps {
                sh 'whoami'
                sh 'echo $PATH'

            }

        }


    }

}

agent01 - is a container with my slave image

But it fails..

Started by user Alexander Mac Replayed #18

git rev-parse --is-inside-work-tree # timeout=10 Setting origin to https://github.com/kondaurov-scala/snippets.git git config remote.origin.url https://github.com/kondaurov-scala/snippets.git # timeout=10 Fetching origin... Fetching upstream changes from origin git --version # timeout=10 using GIT_SSH to set credentials git fetch --tags --progress origin +refs/heads/:refs/remotes/origin/ Seen branch in repository origin/1.1.0 Seen branch in repository origin/master Seen 2 remote branches Obtained Jenkinsfile from 5d917470103056b70398465c0d8a56b127f9e036 [Pipeline] node Running on agent01 in /home/jenkins/workspace/hello_world_1.1.0-NAIZYV5H2QFOICXMDYZJM3A6XMQ4YL2QYG5ZWJDDWOEWF27RHL7Q [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository git config remote.origin.url https://github.com/kondaurov-scala/snippets.git # timeout=10 Fetching without tags Fetching upstream changes from https://github.com/kondaurov-scala/snippets.git git --version # timeout=10 using GIT_SSH to set credentials git fetch --no-tags --progress https://github.com/kondaurov-scala/snippets.git +refs/heads/:refs/remotes/origin/ Checking out Revision 5d917470103056b70398465c0d8a56b127f9e036 (1.1.0) Commit message: "ref" git config core.sparsecheckout # timeout=10 git checkout -f 5d917470103056b70398465c0d8a56b127f9e036 git rev-list 5d917470103056b70398465c0d8a56b127f9e036 # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Agent Setup) [Pipeline] sh [hello_world_1.1.0-NAIZYV5H2QFOICXMDYZJM3A6XMQ4YL2QYG5ZWJDDWOEWF27RHL7Q] Running shell script + docker pull hello-world Using default tag: latest latest: Pulling from library/hello-world 5b0f327be733: Pulling fs layer 5b0f327be733: Verifying Checksum 5b0f327be733: Download complete 5b0f327be733: Pull complete Digest: sha256:b2ba691d8aac9e5ac3644c0788e3d3823f9e97f757f01d2ddc6eb5458df9d801 Status: Downloaded newer image for hello-world:latest [Pipeline] } [Pipeline] // stage [Pipeline] sh [hello_world_1.1.0-NAIZYV5H2QFOICXMDYZJM3A6XMQ4YL2QYG5ZWJDDWOEWF27RHL7Q] Running shell script + docker inspect -f . hello-world . [Pipeline] withDockerContainer agent01 seems to be running inside container c6ad60fd4215a35009accd0be49fbe1ebb583b18b12bf87ba80141b63ae9221a but /home/jenkins/workspace/hello_world_1.1.0-NAIZYV5H2QFOICXMDYZJM3A6XMQ4YL2QYG5ZWJDDWOEWF27RHL7Q could not be found among [/var/run/docker.sock] but /home/jenkins/workspace/hello_world_1.1.0-NAIZYV5H2QFOICXMDYZJM3A6XMQ4YL2QYG5ZWJDDWOEWF27RHL7Q@tmp could not be found among [/var/run/docker.sock] $ docker run -t -d -u 1000:1000 -w /home/jenkins/workspace/hello_world_1.1.0-NAIZYV5H2QFOICXMDYZJM3A6XMQ4YL2QYG5ZWJDDWOEWF27RHL7Q -v /home/jenkins/workspace/hello_world_1.1.0-NAIZYV5H2QFOICXMDYZJM3A6XMQ4YL2QYG5ZWJDDWOEWF27RHL7Q:/home/jenkins/workspace/hello_world_1.1.0-NAIZYV5H2QFOICXMDYZJM3A6XMQ4YL2QYG5ZWJDDWOEWF27RHL7Q:rw,z -v /home/jenkins/workspace/hello_world_1.1.0-NAIZYV5H2QFOICXMDYZJM3A6XMQ4YL2QYG5ZWJDDWOEWF27RHL7Q@tmp:/home/jenkins/workspace/hello_world_1.1.0-NAIZYV5H2QFOICXMDYZJM3A6XMQ4YL2QYG5ZWJDDWOEWF27RHL7Q@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat hello-world [Pipeline] // withDockerContainer [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline java.io.IOException: Failed to run image 'hello-world'. Error: docker: Error response from daemon: oci runtime error: container_linux.go:262: starting container process caused "exec: \"cat\": executable file not found in $PATH". at org.jenkinsci.plugins.docker.workflow.client.DockerClient.run(DockerClient.java:138) at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:179) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:224) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:19) at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/Users/Shared/Jenkins/Home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:128) at org.jenkinsci.plugins.docker.workflow.Docker.node(jar:file:/Users/Shared/Jenkins/Home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:63) at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/Users/Shared/Jenkins/Home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:116) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.DockerPipelineScript.runImage(jar:file:/Users/Shared/Jenkins/Home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy:57) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.configureRegistry(jar:file:/Users/Shared/Jenkins/Home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:68) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.run(jar:file:/Users/Shared/Jenkins/Home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:54) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/Users/Shared/Jenkins/Home/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:60) at cps.transform(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor603.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Finished: FAILURE

2
dockerjenkinsjenkins-pipelinedocker-machine
Error response from daemon: oci runtime error: container_linux.go:262: starting container process caused "exec: \"cat\": executable file not found in $PATH". --< that's your error :) – Sergiu
@Sergiu, i don't think so, i believe this one important: withDockerContainer agent01 seems to be running inside container c6ad60fd4215a35009accd0be49fbe1ebb583b18b12bf87ba80141b63ae9221a but /home/jenkins/workspace/hello_world_1.1.0-NAIZYV5H2QFOICXMDYZJM3A6XMQ4YL2QYG5ZWJDDWOEWF27RHL7Q could not be found among [/var/run/docker.sock] – Alexander Kondaurov
Did you made sure that the users have the right UID and GID inside the container? – Sergiu
@Sergiu, I can ssh to my agent as jenkins and run docker ps for e.g, doesn't this mean that rights are correct? Rights of what? /var/run/docker.sock? – Alexander Kondaurov
Yes the right to run Docker. – Sergiu

2 Answers

1
votes

In my case, that message:

sl01978v-98210312 seems to be running inside container d93e0112b277e94c11bac40af30ddd13ef39cb312caca839497e012cd71bdf86
but /opt/jenkins/workspace/gopvolging_jenkins-somebranch could not be found among [jenkins_slave]
but /opt/jenkins/workspace/gopvolging_jenkins-somebranch@tmp could not be found among [jenkins_slave]

... was caused by the fact that the jenkins slave (which is a docker container) didn't have any volume or mount for the workspace, so the --volume-from option that Jenkins automatically passes in a DiD scenario couldn't be added. It instead tried to pass the workspace in -v, but that wasn't shared with the Jenkins slave container, so it kept waiting endlessly on the first sh command. Adding a volume for the Jenkins workspace solved it:

sl01978v-fbd455e0 seems to be running inside container 0f0090afbcece077f492be581170bf9ffb778c75e23144ccf303e9f47e9cfdd5
$ docker run -t -d -u 1000:1000 -w /opt/jenkins/workspace/gopvolging_jenkins-somebranch --volumes-from 0f0090afbcece077f492be581170bf9ffb778c75e23144ccf303e9f47e9cfdd5 ...

(the magic being --volume-from)

0
votes

Try to link the remote root directory of the slave container to your host /tmp folder. It appears to be /home/jenkins/ in your case

docker run -d -p 2222:22 --name=jenkins_agent -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/home/jenkins