Deploying an existing Spring Cloud Stream application in Spring Cloud Data Flow

Question

I have an application already implemented using Spring Cloud Stream (SCS) with 3 components: 1 source @EnableBinding(Source.class), 1 processor @EnableBinding(Processor.class) and 1 sink @EnableBinding(Sink.class) that I communicate using Apache Kafka binders.

As part of the configuration for these components, I'm using several properties from Spring Cloud Stream, such as the topics to use, the number of partitions, the serializers, the max poll, etc.:

spring:
  application:
    name: myapp

  cloud:
    stream:
      bindings:
        output:
          destination: topic1
          producer:
            partitionCount: 5
            partitionKeyExpression: headers.kafka_messageKey
      kafka:
        binder:
          brokers: 10.138.128.62
          defaultBrokerPort: 9092
          zkNodes: 10.138.128.62
          defaultZkPort: 2181
          requiredAcks: -1
          replicationFactor: 1
          autoCreateTopics: true
          autoAddPartitions: true
        bindings:
          output:
            producer:
              configuration:
                key.serializer: org.apache.kafka.common.serialization.StringSerializer
                value.serializer: org.apache.kafka.common.serialization.ByteArraySerializer

All these properties are defined in an external file 'application.yml' that I indicate at the time of executing the component:

java -jar mycomponent.jar --spring.config.location=/conf/application.yml

Currently, I orchestrate those 3 components "manually", but I would like to use Spring Cloud Data Flow (SCDF) to create a stream and be able to operate them much better.

Based on the SCDF documentation, any SCS application can be straight-forwardly used as an application to be defined in a stream. Besides that, properties for the application can be provided through a external properties file. However, I'm providing my 'application.yml' properties file and it's not working:

stream deploy --name mystream --definition "mysource | myprocessor | mysink' --deploy --propertiesFile /conf/application.yml

After some research, I realized that the documentation states that any property for any application must be passed in this format:

app.<app-name>.<property-name>=<value>

So I have some questions:

Do I have add that "app." to all my existing properties?
Is there any way I can provide something like "--spring.config.location" to my application in SCDF?
If I already provide a "spring.application.name" property in the application.yml, how does it impact SCDF, as I also provide an application name when defining the stream?
If I already provide a "server.port" property in the application.yml, how does it impact SCDF? Will SCDF pick it as the port to use for the application or will it just ignore it?

Thanks in advance for your support.

Ilayaperumal Gopinathan Ilayaperumal Gopinathan · Accepted Answer · 2017-05-26T14:19:10

Do I have to add that "app." to all my existing properties?

Yes. You can have something like this:

app: app-name: spring: cloud: ...

Is there any way I can provide something like "--spring.config.location" to my application in SCDF?

For the stream that is deployed, only --propertiesFile can provide properties at runtime. But, you can still use application specific properties like:

stream deploy mystream --properties "app.*.spring.config.location=configFile" or, different configFiles for each app with the app.app-name prefix.

This way, all the apps that are deployed would get these properties.

If I already provide a "spring.application.name" property in the application.yml, how does it impact SCDF, as I also provide an application name when defining the stream?

Do you use spring.application.name explicitly in your application for some reason. I guess there will be some impact in metrics collector if you change the spring.application.name.

If I already provide a "server.port" property in the application.yml, how does it impact SCDF? Will SCDF pick it as the port to use for the application or will it just ignore it?

It works the same way as Spring Boot property sources precedence. The server.port in your application.yml of your application will get the least precedence over the other property sources that can be set via stream definition/deployment properties.

Deploying an existing Spring Cloud Stream application in Spring Cloud Data Flow

1 Answers