Docker pull by digest fails with Error response from daemon: received unexpected HTTP status: 500 Internal Server Error when Artifactory is used as Docker registry.
The problem causes docker service creation operations to fail if latest tag is used. I have been using Artifactory as Docker registry for about 9 months with no problem. After I upgraded Docker to version 1.13.0 (and to version 1.13.1 later), I started to get No such image: ... errors while creating services with tag latest or with no tag. Artifactory logs showed that pulling is made by digest while docker service is being created from latest images. Upgrading to the latest version of Artifactory(5.0.1) did not worked.
I found an open issue regarding the problem: https://www.jfrog.com/jira/browse/RTFACT-10543
Pulling images by digest might not be a popular operation, but docker service creation from latest image should be very common practice. It is interesting that a few complaints exist about the problem. That is why I started to think whether the problem is somehow related with my Artifactory configuration.
Docker pull from DockerHub - Works
[myuser@rose3 ~]$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
4b0bc1c4050b: Pull complete
Digest: sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
Status: Downloaded newer image for busybox:latest
Docker pull by digest from DockerHub - Works
[myuser@rose3 ~]$ docker pull busybox@sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e: Pulling from library/busybox
4b0bc1c4050b: Pull complete
Digest: sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
Status: Downloaded newer image for busybox@sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
Docker pull from Artifactory - Works
[myuser@rose3 ~]$ docker pull artifactory.mycompany.com/busybox
Using default tag: latest
latest: Pulling from busybox
4b0bc1c4050b: Pull complete
Digest: sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
Status: Downloaded newer image for artifactory.mycompany.com/busybox:latest
Apache - access_log
10.74.200.56 - - [27/Feb/2017:09:51:27 +0300] "GET /v2/ HTTP/1.1" 401 77 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))"
10.74.200.56 - - [27/Feb/2017:09:51:27 +0300] "GET /artifactory/api/docker/docker/v2/token?account=akumlali&scope=repository%3Abusybox%3Apull&service=artifactory.mycompany.com HTTP/1.1" 200 103 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))"
10.74.200.56 - - [27/Feb/2017:09:51:32 +0300] "GET /v2/busybox/manifests/latest HTTP/1.1" 200 527 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))"
10.74.200.56 - - [27/Feb/2017:09:51:35 +0300] "GET /v2/busybox/blobs/sha256:7968321274dc6b6171697c33df7815310468e694ac5be0ec03ff053bb135e768 HTTP/1.1" 200 1465 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))"
10.74.200.56 - - [27/Feb/2017:09:51:35 +0300] "GET /v2/busybox/blobs/sha256:4b0bc1c4050b03c95ef2a8e36e25feac42fd31283e8c30b3ee5df6b043155d3c HTTP/1.1" 200 677628 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))"
Artifactory - artifactory.log
2017-02-27 09:51:32,352 [http-nio-8081-exec-46] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'busybox' and tag 'latest'
2017-02-27 09:51:35,684 [http-nio-8081-exec-46] [INFO ] (o.a.r.HttpRepo :420) - registry-1.docker.io downloading https://registry-1.docker.io/v2/library/busybox/manifests/latest 527 bytes
2017-02-27 09:51:35,691 [http-nio-8081-exec-46] [INFO ] (o.a.r.HttpRepo :433) - registry-1.docker.io downloaded https://registry-1.docker.io/v2/library/busybox/manifests/latest 527 bytes at 89.81 KB/sec
2017-02-27 09:51:35,712 [http-nio-8081-exec-44] [INFO ] (o.j.r.d.v.r.h.DockerV2LocalRepoHandler:103) - Fetching docker blob 'sha256:7968321274dc6b6171697c33df7815310468e694ac5be0ec03ff053bb135e768' from repo 'docker-local2'
2017-02-27 09:51:35,712 [http-nio-8081-exec-9] [INFO ] (o.j.r.d.v.r.h.DockerV2LocalRepoHandler:103) - Fetching docker blob 'sha256:4b0bc1c4050b03c95ef2a8e36e25feac42fd31283e8c30b3ee5df6b043155d3c' from repo 'docker-local2'
2017-02-27 09:51:35,728 [http-nio-8081-exec-9] [INFO ] (o.a.a.d.r.v.DockerV2RemoteRepoHandler:133) - Fetching docker blob 'sha256:4b0bc1c4050b03c95ef2a8e36e25feac42fd31283e8c30b3ee5df6b043155d3c'
2017-02-27 09:51:35,728 [http-nio-8081-exec-44] [INFO ] (o.a.a.d.r.v.DockerV2RemoteRepoHandler:133) - Fetching docker blob 'sha256:7968321274dc6b6171697c33df7815310468e694ac5be0ec03ff053bb135e768'
Docker pull by digest from Artifactory - Does not work!
[myuser@rose3 ~]$ docker pull artifactory.mycompany.com/busybox@sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e
Error response from daemon: received unexpected HTTP status: 500 Internal Server Error
Apache - access_log
10.74.200.56 - - [27/Feb/2017:09:52:13 +0300] "GET /v2/ HTTP/1.1" 401 77 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))"
10.74.200.56 - - [27/Feb/2017:09:52:13 +0300] "GET /artifactory/api/docker/docker/v2/token?account=akumlali&scope=repository%3Abusybox%3Apull&service=artifactory.mycompany.com HTTP/1.1" 200 103 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))"
10.74.200.56 - - [27/Feb/2017:09:52:13 +0300] "GET /v2/busybox/manifests/sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e HTTP/1.1" 500 149 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))"
Artifactory - artifactory.log
2017-02-27 09:52:13,232 [http-nio-8081-exec-48] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'busybox' and tag 'sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e'
2017-02-27 09:52:13,233 [http-nio-8081-exec-48] [INFO ] (o.j.r.d.v.r.h.DockerV2LocalRepoHandler:533) - Fetching docker manifest for repo 'busybox' and digest 'sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e' in repo 'docker-local2'
2017-02-27 09:52:13,237 [http-nio-8081-exec-48] [ERROR] (o.a.r.c.e.GlobalExceptionMapper:48) - null
java.lang.NullPointerException: null
UPDATE 1:
I upgraded to the latest version of Docker (17.03.0-ce) and Artifactory (5.1.0) with no luck.
Docker Version
[myuser@rose3 myuser]$ sudo docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 17.03.0-ce
Storage Driver: devicemapper
Pool Name: docker-253:3-33595530-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 16.91 MB
Data Space Total: 107.4 GB
Data Space Available: 9.287 GB
Metadata Space Used: 585.7 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-327.36.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.51 GiB
Name: rose3
ID: 3P6Q:V4IC:C4RZ:MIZP:WT2R:PW4H:QSZC:G3MU:KUEY:3HSM:FX3U:TDO6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Apache - access_log
10.74.200.64 - - [03/Mar/2017:14:07:59 +0300] "GET /api/docker/docker/v2/ HTTP/1.1" 401 77 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/60ccb22 kernel/3.10.0-327.36.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \\(linux\\))"
10.74.200.64 - - [03/Mar/2017:14:07:59 +0300] "GET /v2/ HTTP/1.1" 401 77 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/60ccb22 kernel/3.10.0-327.36.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \\(linux\\))"
10.74.200.64 - - [03/Mar/2017:14:07:59 +0300] "GET /api/docker/docker/v2/token?scope=repository%3Abusybox%3Apull&service=artifactory.mycompany.com%3A443 HTTP/1.1" 200 103 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/60ccb22 kernel/3.10.0-327.36.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \\(linux\\))"
10.74.200.64 - - [03/Mar/2017:14:07:59 +0300] "GET /api/docker/docker/v2/busybox/manifests/sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e HTTP/1.1" 500 149 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/60ccb22 kernel/3.10.0-327.36.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \\(linux\\))"
10.74.200.64 - - [03/Mar/2017:14:07:59 +0300] "GET /v2/busybox/manifests/sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e HTTP/1.1" 500 149 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/60ccb22 kernel/3.10.0-327.36.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \\(linux\\))"
Artifactory - artifactory.log
2017-03-03 13:56:28,267 [http-nio-8081-exec-9] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'busybox' and tag 'sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e'
2017-03-03 13:56:28,268 [http-nio-8081-exec-9] [INFO ] (o.j.r.d.v.r.h.DockerV2LocalRepoHandler:533) - Fetching docker manifest for repo 'busybox' and digest 'sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e' in repo 'docker-local'
2017-03-03 13:56:28,269 [http-nio-8081-exec-9] [ERROR] (o.a.r.c.e.GlobalExceptionMapper:48) - null
java.lang.NullPointerException: null
at org.artifactory.addon.docker.rest.v2.repo.virtual.DockerV2VirtualRepoHandler.execute(DockerV2VirtualRepoHandler.java:178) ~[artifactory-addon-docker-5.1.0.jar:na]
at org.artifactory.addon.docker.rest.v2.repo.virtual.DockerV2VirtualRepoHandler.getManifest(DockerV2VirtualRepoHandler.java:106) ~[artifactory-addon-docker-5.1.0.jar:na]
at org.jfrog.repomd.docker.v2.rest.DockerV2Resource.getManifest(DockerV2Resource.java:81) ~[docker-4.3.21.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [servlet-api.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) [catalina.jar:8.0.39]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.39]
at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:185) [artifactory-web-application-5.1.0.jar:na]
at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:94) [artifactory-web-application-5.1.0.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.39]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.39]
at org.artifactory.webapp.servlet.authentication.ArtifactoryAuthenticationFilterChain.lambda$1(ArtifactoryAuthenticationFilterChain.java:136) [artifactory-web-application-5.1.0.jar:na]
at org.artifactory.webapp.servlet.authentication.PropsAuthenticationFilter.doFilter(PropsAuthenticationFilter.java:131) ~[artifactory-web-application-5.1.0.jar:na]
at org.artifactory.webapp.servlet.authentication.ArtifactoryAuthenticationFilterChain.doFilter(ArtifactoryAuthenticationFilterChain.java:172) [artifactory-web-application-5.1.0.jar:na]
at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:281) ~[artifactory-web-application-5.1.0.jar:na]
at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:205) ~[artifactory-web-application-5.1.0.jar:na]
at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:165) ~[artifactory-web-application-5.1.0.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.39]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.39]
at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:67) ~[artifactory-web-application-5.1.0.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.39]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.39]
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164) ~[spring-session-1.2.2.RELEASE.jar:na]
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80) ~[spring-session-1.2.2.RELEASE.jar:na]
at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62) ~[artifactory-web-application-5.1.0.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.39]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.39]
at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:116) ~[artifactory-web-application-5.1.0.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.39]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.39]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) ~[catalina.jar:8.0.39]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) ~[catalina.jar:8.0.39]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) ~[catalina.jar:8.0.39]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) ~[catalina.jar:8.0.39]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) ~[catalina.jar:8.0.39]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:509) ~[catalina.jar:8.0.39]
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1104) ~[tomcat-coyote.jar:8.0.39]
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684) ~[tomcat-coyote.jar:8.0.39]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520) ~[tomcat-coyote.jar:8.0.39]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476) ~[tomcat-coyote.jar:8.0.39]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_121]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-util.jar:8.0.39]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
UPDATE 2
The problem still exists with Artifactory 5.1.2.
UPDATE 3
It seems that Artifactory does not handle recent changes of Docker regarding image tags. Simply put:
docker service create ...pulls images by digest (foo@sha256:aabbff11)- When pulling an image by digest (foo@sha256:aabbff11), no tag is created, because the same digest can be shared by different tags.
- Artifactory expects a tag and throws NPE while creating services or pulling images by digest.
There are active discussions in Docker community:
- [1.13] starting swarm services results in images with
<none>tag #28908 - docker stack deploy creates no tags #31574
- Docker SHA-based (digest) pulls should respect and preserve image tag(s) #31656
Here are the simple test cases showing that docker run --name=busybox busybox:1.26.2 sleep 60 creates tag 1.26.2, while docker service create --name=busybox busybox:1.26.2 sleep 60 does not. Also test cases show that Artifactory does not handle the cases where the tag is <none>.
Case 1
[myuser@rose3 ~]$ docker run --name=busybox busybox:1.26.2 sleep 60
Unable to find image 'busybox:1.26.2' locally
1.26.2: Pulling from library/busybox
7520415ce762: Pull complete
Digest: sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f
Status: Downloaded newer image for busybox:1.26.2
[myuser@rose3 ~]$ docker images busybox
REPOSITORY TAG IMAGE ID CREATED SIZE
busybox 1.26.2 00f017a8c2a6 4 days ago 1.11 MB
Case 2
[myuser@rose2 ~]$ docker run --name=busybox artifactory.mycompany.com/busybox:1.26.2 sleep 60
Unable to find image 'artifactory.mycompany.com/busybox:1.26.2' locally
1.26.2: Pulling from busybox
7520415ce762: Pull complete
Digest: sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f
Status: Downloaded newer image for artifactory.mycompany.com/busybox:1.26.2
[myuser@rose2 ~]$ docker images artifactory.mycompany.com/busybox
REPOSITORY TAG IMAGE ID CREATED SIZE
artifactory.mycompany.com/busybox 1.26.2 00f017a8c2a6 4 days ago 1.11 MB
Case 3
[myuser@rose1 ~]$ docker service create --name=busybox busybox:1.26.2 sleep 60
9wi3h70is2p64andg943jihf8
[myuser@rose1 ~]$ docker service ps busybox
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
vhqgxhcrpesu busybox.1 busybox:1.26.2 rose3 Running Running 1 second ago
[myuser@rose3 ~]$ docker images busybox
REPOSITORY TAG IMAGE ID CREATED SIZE
busybox <none> 00f017a8c2a6 4 days ago 1.11 MB
Case 4
[myuser@rose1 ~]$ docker service create --name=busybox artifactory.mycompany.com/busybox:1.26.2 sleep 60
phzmsqktm9ybt6fzs4d5mrd8e
[myuser@rose1 ~]$ docker service ps busybox
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
sppf72atj1z9 busybox.1 artifactory.mycompany.com/busybox:1.26.2 rose3 Ready Rejected 2 seconds ago "No such image: artifactory.my…"
txq73aq1r06g \_ busybox.1 artifactory.mycompany.com/busybox:1.26.2 rose3 Shutdown Rejected 7 seconds ago "No such image: artifactory.my…"
f9c9evp74059 \_ busybox.1 artifactory.mycompany.com/busybox:1.26.2 rose3 Shutdown Rejected 12 seconds ago "No such image: artifactory.my…"
jb16jyorggeb \_ busybox.1 artifactory.mycompany.com/busybox:1.26.2 rose3 Shutdown Rejected 12 seconds ago "No such image: artifactory.my…"
[myuser@rose1 ~]$
[myuser@rose1 ~]$ docker inspect sppf72atj1z9
[
{
"ID": "sppf72atj1z9xpgarj9jp6yw8",
"Version": {
"Index": 779121
},
"CreatedAt": "2017-03-14T11:52:36.347033515Z",
"UpdatedAt": "2017-03-14T11:52:41.585341679Z",
"Spec": {
"ContainerSpec": {
"Image": "artifactory.mycompany.com/busybox:1.26.2@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f",
"Args": [
"sleep",
"60"
],
"DNSConfig": {}
},
"Resources": {
"Limits": {},
"Reservations": {}
},
"RestartPolicy": {
"Condition": "any",
"MaxAttempts": 0
},
"Placement": {},
"ForceUpdate": 0
},
"ServiceID": "phzmsqktm9ybt6fzs4d5mrd8e",
"Slot": 1,
"NodeID": "rdg9afjb1ghmgw6rdgfrw7dq3",
"Status": {
"Timestamp": "2017-03-14T11:52:36.667378877Z",
"State": "rejected",
"Message": "preparing",
"Err": "No such image: artifactory.mycompany.com/busybox@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f",
"ContainerStatus": {},
"PortStatus": {}
},
"DesiredState": "shutdown"
}
]
Case 5
[myuser@rose3 ~]$ docker run --name=busybox busybox@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f sleep 60
Unable to find image 'busybox@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f' locally
sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f: Pulling from library/busybox
7520415ce762: Pull complete
Digest: sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f
Status: Downloaded newer image for busybox@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f
[myuser@rose3 ~]$ docker images busybox
REPOSITORY TAG IMAGE ID CREATED SIZE
busybox <none> 00f017a8c2a6 4 days ago 1.11 MB
Case 6
[myuser@rose3 ~]$ docker run --name=busybox artifactory.mycompany.com/busybox@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f sleep 60
Unable to find image 'artifactory.mycompany.com/busybox@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f' locally
docker: Error response from daemon: received unexpected HTTP status: 500 Internal Server Error.
See 'docker run --help'.
[myuser@rose3 ~]$
artifactory.log
2017-03-14 14:50:04,388 [http-nio-8081-exec-114] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'busybox' and tag 'sha256:32f093055929dbc23dec4d03e09dfe971f5
973a9ca5cf059cbfb644c206aa83f'
2017-03-14 14:50:04,388 [http-nio-8081-exec-114] [INFO ] (o.j.r.d.v.r.h.DockerV2LocalRepoHandler:533) - Fetching docker manifest for repo 'busybox' and digest 'sha256:32f093055929dbc23dec4d03e09dfe971f59
73a9ca5cf059cbfb644c206aa83f' in repo 'docker-local2'
2017-03-14 14:50:04,402 [http-nio-8081-exec-114] [ERROR] (o.a.r.c.e.GlobalExceptionMapper:48) - null
java.lang.NullPointerException: null
at org.artifactory.addon.docker.rest.v2.repo.virtual.DockerV2VirtualRepoHandler.execute(DockerV2VirtualRepoHandler.java:178) ~[artifactory-addon-docker-5.1.2.jar:na]
at org.artifactory.addon.docker.rest.v2.repo.virtual.DockerV2VirtualRepoHandler.getManifest(DockerV2VirtualRepoHandler.java:106) ~[artifactory-addon-docker-5.1.2.jar:na]
at org.jfrog.repomd.docker.v2.rest.DockerV2Resource.getManifest(DockerV2Resource.java:81) ~[docker-4.3.21.jar:na]
UPDATE 4:
RTFACT-10543 fixed with Artifactory 5.2.1 release. I confirmed Artifactory happily handles null tag now.
[myuser@rose3 ~]$ docker pull artifactory.mycompany.com/busybox@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f
sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f: Pulling from busybox
7520415ce762: Pull complete
Digest: sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f
Status: Downloaded newer image for artifactory.mycompany.com/busybox@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f
Service creation problem has also been resolved with this fix: https://github.com/moby/moby/issues/28908
