Spring Cloud DataFlow Kubernetes Server does not work with valid docker URIs

Question

I am in the process of upgrading a Spring Cloud DataFlow Kubernetes Server deployment from 1.2.2.RELEASE to 1.3.1.RELEASE, testing it with minikube.

After fixing invalid docker URIs (docker: to docker://) and successfully seeing apps registered in the shell, the deployment to the kubernetes server ultimately fails. With a simple stream:

mysource | myprocessor | rabbitsink

the Spring Boot sink app does successfully deploy and start up according to the pod's logs. Here's the successful registration of that app:

app register --name rabbitsink     --type sink       --uri docker://192.168.99.100:80/myrepo/rabbitsink:current
Successfully registered application 'sink:rabbitsink'

Despite this, the subsequent output from the dataflow shell is:

Command failed org.springframework.cloud.dataflow.rest.client.DataFlowClientException: Invalid docker resource URI: docker:192.168.99.100:80/myrepo/rabbitsink:current

and none of the stream's other apps starts up.

Here's the stacktrace from the log:

2018-03-12 13:17:16.470  INFO 1 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 9393 (http)
2018-03-12 13:17:16.516  INFO 1 --- [           main] o.s.c.d.s.k.KubernetesDataFlowServer     : Started KubernetesDataFlowServer in 54.261 seconds (JVM running for 59.213)
2018-03-12 13:17:49.724  INFO 1 --- [nio-9393-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring FrameworkServlet 'dispatcherServlet'
2018-03-12 13:17:53.604  INFO 1 --- [nio-9393-exec-1] .s.c.d.s.s.i.AppDeploymentRequestCreator : Creating resource with [docker://192.168.99.100:80/myrepo/rabbitsink:current] for application [rabbitsink]
2018-03-12 13:17:53.618  INFO 1 --- [nio-9393-exec-1] .s.c.d.s.s.i.AppDeploymentRequestCreator : Creating resource with [docker://192.168.99.100:80/myrepo/myprocessor:current] for application [myprocessor]
2018-03-12 13:17:53.624  INFO 1 --- [nio-9393-exec-1] .s.c.d.s.s.i.AppDeploymentRequestCreator : Creating resource with [docker://192.168.99.100:80/myrepo/mysource:current] for application [mysource]
2018-03-12 13:17:53.629  INFO 1 --- [nio-9393-exec-1] o.s.c.d.s.s.AppDeployerStreamDeployer    : Deploying application named [rabbitsink] as part of stream named [mystream] with resource URI [docker:192.168.99.100:80/myrepo/rabbitsink:current]
2018-03-12 13:17:54.833  INFO 1 --- [nio-9393-exec-1] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker image: 192.168.99.100:80/myrepo/rabbitsink:current
2018-03-12 13:17:54.837  INFO 1 --- [nio-9393-exec-1] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker entry point style: exec
2018-03-12 13:17:58.157  INFO 1 --- [nio-9393-exec-1] o.s.c.d.s.s.AppDeployerStreamDeployer    : Deploying application named [myprocessor] as part of stream named [mystream] with resource URI [docker:192.168.99.100:80/myrepo/myprocessor:current]
2018-03-12 13:17:58.935  INFO 1 --- [nio-9393-exec-1] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker image: 192.168.99.100:80/myrepo/myprocessor:current
2018-03-12 13:17:58.935  INFO 1 --- [nio-9393-exec-1] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker entry point style: exec
2018-03-12 13:18:04.523  INFO 1 --- [nio-9393-exec-1] o.s.c.d.s.s.AppDeployerStreamDeployer    : Deploying application named [mysource] as part of stream named [mystream] with resource URI [docker:192.168.99.100:80/myrepo/mysource:current]
2018-03-12 13:18:05.380  INFO 1 --- [nio-9393-exec-1] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker image: 192.168.99.100:80/myrepo/mysource:current
2018-03-12 13:18:05.419  INFO 1 --- [nio-9393-exec-1] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker entry point style: exec
2018-03-12 13:18:17.497 ERROR 1 --- [nio-9393-exec-1] o.s.c.d.s.c.RestControllerAdvice         : Caught exception while handling a request

java.lang.IllegalArgumentException: Invalid docker resource URI: docker:192.168.99.100:80/myrepo/rabbitsink:current
    at org.springframework.util.Assert.isTrue(Assert.java:92) ~[spring-core-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.cloud.dataflow.registry.support.ResourceUtils.formatDockerResource(ResourceUtils.java:173) ~[spring-cloud-dataflow-registry-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
    at org.springframework.cloud.dataflow.registry.support.ResourceUtils.getResourceVersion(ResourceUtils.java:147) ~[spring-cloud-dataflow-registry-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
    at org.springframework.cloud.dataflow.server.stream.AppDeployerStreamDeployer.deployStream(AppDeployerStreamDeployer.java:141) ~[spring-cloud-dataflow-server-core-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
    at org.springframework.cloud.dataflow.server.service.impl.AppDeployerStreamService.doDeployStream(AppDeployerStreamService.java:88) ~[spring-cloud-dataflow-server-core-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
    at org.springframework.cloud.dataflow.server.service.impl.AbstractStreamService.deployStream(AbstractStreamService.java:85) ~[spring-cloud-dataflow-server-core-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
    at org.springframework.cloud.dataflow.server.controller.StreamDeploymentController.deploy(StreamDeploymentController.java:153) ~[spring-cloud-dataflow-server-core-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111-internal]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111-internal]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111-internal]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111-internal]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) ~[spring-webmvc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) ~[spring-webmvc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) [spring-boot-1.5.9.RELEASE.jar!/:1.5.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) [spring-boot-actuator-1.5.9.RELEASE.jar!/:1.5.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.2.3.RELEASE.jar!/:4.2.3.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) [spring-boot-actuator-1.5.9.RELEASE.jar!/:1.5.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111-internal]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111-internal]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111-internal]

Answer 1

We have recently discussed a similar issue in spring-cloud/spring-cloud-dataflow#2065.

As highlighted in the comment, the Docker resource URIs are expected to be of a certain format starting from 1.3 release.

In your case, the Docker URI should have been:

dataflow:>app register --name rabbitsink --type sink --uri docker:myrepo/rabbitsink:current

If the above format is not followed, the Shell would fail-fast with a validation message. The stream won't be deployed; no pods should show up for the apps either. Here's the validation message on my minikube environment.

Command failed org.springframework.cloud.dataflow.rest.client.DataFlowClientException: Invalid docker resource URI: docker:192.168.99.100:80/myrepo/rabbitsink:current

I'd recommend starting fresh. You could also use the helm-chart (Docs for Minikube is in PR) to simplify the provisioning experience.

Answer 2

It is a bug we need to address from what I can tell. The test

    @Test
public void testDockerResourceProcessingSO() {
    DockerResource dockerResource = new DockerResource("192.168.99.100:80/myrepo/rabbitsink:current");
    assertThat(ResourceUtils.getResourceWithoutVersion(dockerResource)).isEqualTo("docker:192.168.99.100:80/myrepo/rabbitsink");
    assertThat(ResourceUtils.getResourceVersion(dockerResource)).isEqualTo("current");
}

Fails. Due to this line of code

            String dockerResourceUri = dockerResource.getURI().toString();
        Assert.isTrue(StringUtils.countOccurrencesOf(dockerResourceUri, ":") == 2,
                "Invalid docker resource URI: " + dockerResourceUri)

and the dockerResourceUri is docker:192.168.99.100:80/myrepo/rabbitsink:current

There is also going to be another issue down the line if using skipper, in that the version needs to follow 'semver' format and 'current' isn't going to be valid. We can look to better validate the docker format and get a fix in asap.

After we fix that, if you are using skipper, then skipper won't like 'current' as a version name, it needs to be a real version number. Not yet sure if we can do anything to accommodate a 'dev workflow' without version numbers when the whole point of skipper was to manage version updates. Maybe we need to allow 'current', 'latest', 'SNAPSHOT' as allowed versions and always redeploy those apps when and update is request.

Answer 3

This has been fixed in the 1.4.0.RELEASE version of Spring Cloud Dataflow and is working nicely with the 1.4.0.RELEASE versions of the local dataflow server, the kubernetes dataflow server and dataflow shell. It works with Spring Cloud Edgware.SR2 and Edgware.SR3.