spring-cloud-starter: stream-processor-tasklaunchrequest-transform ignores local maven settings

Question

In my cloud stream application I've used the spring-cloud-starter-stream-processor-tasklaunchrequest-transform starter app as task-processor.

The starter is locally built and installed in local Maven repository.

Commands used to start my local Dataflow Server and DF Shell, and then register my apps and create-cum-deploy the stream are as below:-

• Install the task-launcher app in local maven - mvn clean install
• Install the task-processor app in local maven - mvn clean install

• Install the task app in local maven - mvn clean install

• Build Dataflow Server Local - mvn clean package (with changed remoteRepo location to our enterprise maven repo)

• Start DF Server Local - mvn spring-boot:run

• Build Dataflow Shell - mvn clean package

• Start DF Shell - mvn spring-boot:run

• Import out of box starter apps with Rabbit binding stream-applications-rabbit-maven

• Register apps -

  1. app register --name task-processor --type processor --uri maven://org.springframework.cloud.stream.app:spring-cloud-starter-stream-processor-tasklaunchrequest-transform:jar:1.1.0.BUILD-SNAPSHOT
  2. app register --name task-launcher-local --type sink --uri maven://org.springframework.cloud.stream.app:task-launcher-local-sink-rabbit:jar:1.0.4.RELEASE

• Create Stream - stream create foo --definition "rabbit --rabbit.queues=cloud-stream-source --rabbit.requeue=true --spring.rabbitmq.host=localhost --spring.rabbitmq.port=5672 --spring.cloud.stream.bindings.output.contentType='text/plain' | task-processor --uri=maven://com.example:thumbnail-generator:0.0.1-SNAPSHOT | task-launcher-local" --deploy

However the deploy of the stream is failing due to startup failure of the task-processor. The error logged in the processor logs is:

no main manifest attribute, in /Users/abc/.m2/repository/org/springframework/cloud/stream/app/spring-cloud-starter-stream-processor-tasklaunchrequest-transform/1.1.0.BUILD-SNAPSHOT/spring-cloud-starter-stream-processor-tasklaunchrequest-transform-1.1.0.BUILD-SNAPSHOT.jar

Now there are 2 points to raise:

1. The maven repo location mentioned in the logs is default location for MacOS and not the one setup in my local maven configurations (/Users/abc/myrepo/.m2/repository)
2. The error is valid for the built jar file of starter app - spring-cloud-starter-stream-processor-tasklaunchrequest-transform. And that confuses me if any customisation is required to make it runnable and usable for my stream app.

I've tried changing the starter app to a runnable spring boot app, and installed that version in my local repo but unless the application looks at correct local repo (and not at the default one), I can't use it.

I believe it's an issue with the app and it lacks the visibility of the local maven settings, hence going by default values.

Any inputs will be of help.

Answer 1

Register apps - app register --name task-processor --type processor --uri maven://io.spring.cloud:taskprocessor:jar:1.1.0.BUILD-SNAPSHOT

According to this, you're registering the sample taskprocessor application. Though there's nothing wrong in trying it out, we would recommend using the standardized application and if there any missing features in it, please file a GH issue for it.

However the deploy of the stream is failing due to startup failure of the task-processor.

If you're in fact trying to use the standardized application, you're application registration command should be something like:

app register --name task-processor --type processor --uri maven://org.springframework.cloud.stream.app:tasklaunchrequest-transform-processor-rabbit:1.1.0.BUILD-SNAPSHOT  --force

The maven repo location mentioned in the logs is default location for MacOS and not the one setup in my local maven configurations (/Users/abc/myrepo/.m2/repository)

SCDF uses Aether library to resolve and download maven artifacts. By default, the requested application is checked in local maven repository first and if it isn't present, it is then downloaded from the remote repository. This is no different than any other maven workflow.

The error is valid for the built jar file of starter app - spring-cloud-starter-stream-processor-tasklaunchrequest-transform. And that confuses me if any customisation is required to make it runnable and usable for my stream app

The OOTB applications are utility applications. To the most part, they should satisfy common requirements. If there's a need to adjust the generic behavior, you could then extend them.

With that said, perhaps you could try running the locally built application(s) via java -jar to confirm whether the application bootstraps successfully and then register it to SCDF. You could also enable DEBUG level logs at each application for troubleshooting.