Spring Cloud Data Flow: task can't start on Kubernetes cluster

2
votes

I've deployed Spring Cloud Data Flow server on a local Kubernetes cluster. All seems fine. Then I create an App of type Task, giving the URL of a Spring-Boot JAR. Then I create a task 'definition' and launch it. The task definition hangs in status 'launching'.

Here are my findings:

  1. Looking at Kubernetes, I see the a pod corresponding to the task correctly created but failing to start, with status ImagePullBackOff

  2. This pod is configured with image: /tmp/deployer-resource-cache5494152820122807128/https-60030cec0dd24157b95f59cd3e5b0819916e4adc, and the logs show the message:

    Failed to pull image "/tmp/deployer-resource-cache5494152820122807128/https-60030cec0dd24157b95f59cd3e5b0819916e4adc": couldn't parse image reference "/tmp/deployer-resource-cache5494152820122807128/https-60030cec0dd24157b95f59cd3e5b0819916e4adc": invalid reference format

  3. I connect to the SCDF server pod shell, check out the /tmp folder, and see the deployer-resource-cache5494152820122807128 folder there.

My understanding is SCDF creates a temp image to be executed in the Kubernetes pod, but this image is created inside the scdf server pod, so it's obviously not available from the task pod.

My question is how is this supposed to work?

In my opinion the image should be pushed to a registry, or stored on a shared volume somehow, but I didn't find anything on the topic in the documentation. Any idea or suggestion would be appreciated.

1
kubernetesspring-cloud-dataflow

1 Answers

3
votes

What you are trying to do will not work for Spring Cloud Dataflow for Kubernetes. In the Kubernetes implementaton, only Docker images are supported as a deployment artifact.

Currently this fact is not explicitly mentioned in the documentation. Check out this and this issue