How to reference local Kafka and Zookeeper config on Spring Cloud Dataflow “Cloudfoundry” server start

Question

Here's is what I have successfully done so far on SCDF Local Server

1. I have successfully deployed SCDF server on my local and also I have used Kafka and Zookeeper config parameters with it i.e

mymac$ java -jar spring-cloud-dataflow-server-local-1.3.0.RELEASE.jar --spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.brokers=localhost:9092 --spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.zkNodes=localhost:2181

I was able to create my stream

1. ingest = producer-app > :broker1

2. filter = :broker1 > filter-app > :broker2

---

Now I need help to do the exact same thing on PCFDev

1. I have my PCFDEv running
2. I have to deploy SCDF-Cloudfoundry jar with my local kafka and zookeeper parameters to pcfDev but when I do the following steps it gives me an error that its

1.1) cf push -f manifest-scdf.yml --no-start -p /XXX/XXX/XXX/spring-cloud-dataflow-server-cloudfoundry-1.3.0.BUILD-SNAPSHOT.jar -k 1500M

this runs good...no problem. but 1.2

1.2) cf start dataflow-server --spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.brokers=host.pcfdev.io:9092 --spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.zkNodes=host.pcfdev.io:2181

gives me this error:--

Incorrect Usage: unknown flag `spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.brokers'

below is my manifest-scdf.yml file

---
instances: 1                        
memory: 2048M                         
applications:                         
  - name: dataflow-server                   
    host: dataflow-server                  
    services:                       
      - redis                      
      - rabbit                         
    env:                          
      SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_URL: https://api.local.pcfdev.io
      SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_ORG: pcfdev-org
      SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_SPACE: pcfdev-space
      SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_DOMAIN: local.pcfdev.io
      SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_USERNAME: admin
      SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_PASSWORD: admin
      SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_SKIP_SSL_VALIDATION: true
      SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_SERVICES: rabbit
      MAVEN_REMOTE_REPOSITORIES_REPO1_URL: https://repo.spring.io/libs-snapshot
      SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_DISK: 512
      SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_BUILDPACK: java_buildpack
      spring.cloud.deployer.cloudfoundry.stream.memory: 400         
      spring.cloud.dataflow.features.tasks-enabled: true            
      spring.cloud.dataflow.features.streams-enabled: true        

Please help me. Thank you.

Answer 1

There are few options to supply Kafka credentials to Stream-apps in PCF.

1. Kafka CUPs

This option allows you to create CUPs for an external Kafka-service. While deploying the stream, you can then supply the coordinates to each application either individually as described in the docs or you can supply them as global properties for all the stream-apps deployed by the SCDF-server.

2. Inline properties

Instead of extracting from CUPs, you can also directly supply the HOST/PORT while deploying the stream. Again, this can be applied globally, too.

stream deploy myTest --properties "app.*.spring.cloud.stream.kafka.binder.brokers=<HOST>:9092,app.*.spring.cloud.stream.kafka.binder.zkNodes=<HOST>:2181

Note: The HOST must be reachable for the stream-apps; o'wise, it ill continue to connect to localhost and potentially fail since the apps are running inside a VM.

Answer 2

The error you're seeing is coming from the CF CLI, it's interpreting those (I'm assuming environment) variables you're providing as flags to the cf start command and failing.

You could either provide them in your manifest.yml or set their values manually using the CLI's cf set-env command by doing something like this:

cf set-env dataflow-server spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.brokers host.pcfdev.io:9092

cf set-env dataflow-server spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.zkNodes host.pcfdev.io:2181

After you've set them they should be picked up when you run cf start dataflow-server.

Relevant CLI docs: http://cli.cloudfoundry.org/en-US/cf/set-env.html