Discovering virtualized URL for http app deployed in YARN with spring cloud dataflow

Question

I successfully deployed a stream in to YARN created using SCDF which looks something like -- stream create --name httpTest--definition "http --server.port=9001|....

I was able to scale the http source apps with clustermodify to multiple instances. Does SCDF writes a virtualized / static / proxy URL somewhere in the YARN Registry (see here) so that upstream applications can bind to? How another application outside Yarn can post to above http server with port 9001 that too scaled horizontally?

If i deploy a spring boot web application in docker-swarm and scale it, docker swarm manages to distribute the load among all applications. How does it happen in case of YARN & SCDF in particualr?

When I invoke following in dataflow shell -- it fails too..

http post --target http://localhost:9001 --data empty

Error Message :

Failed to access http endpoint http://localhost:9001

All other variations of replacing localhost with hostname/ipaddress are failing too?

Answer 1

I am answering my own question.

Alternative Design -- Use Kafka destination instead of HTTP Source since HTTP source's URL isn't guaranteed to be unique once deployed in yarn containers ( -- which can be published to a consuming application. ) Alternatively, we will need to patch the OOTB HTTP Source app to enlist itself (at boot time) with Spring's Eureka server implementation and use a gateway as a smart proxy.