Unsuccessful deployment on Spring Cloud Dataflow with Apache YARN

0
votes

I have installed a single-node Apache YARN with Kafka-Zookeeper and Spring Cloud Dataflow 1.0.3.

All is working fine, but when I made some deployment examples, like:

stream create --name "ticktock" --definition "time | hdfs --rollover=100" --deploy

http --port=8000 | log --name=logtest --level=INFO

The stream's status never stays on "deployed". It keeps changing on "undeployed"->"parcial"->"deployed" in a constant loop.

However, the YARN application deploys successfully, but it's like the comunication between Spring Cloud Dataflow server instance and Apache Hadoop it's constantly failing.

What can be the possible issue for this?

Thanks in advance!

2
hadoophadoop-yarnspring-cloud-dataflow
When it comes to a loop of deployment failures, the usual suspect is around the memory allocated for app-master and yarn containers. How are you provisioning SCDF server on YARN? Are you using the Ambari plugin? (if not, why?). Please review this section on memory requirements. There are global settings that you can override, too. – Sabby Anandan

2 Answers

0
votes

Well, there's not a lot of information to go on here, but you may want to check and make sure you have the necessary base directory created in HDFS and that your Yarn user has read/write permission to it.

spring.cloud.deployer.yarn.app.baseDir=/dataflow

0
votes

Thanks to all for your answers! Been busy with other high-priority projects, but I was able to build all using Ambari (including the plugin mentioned by Sabby Adannan). Now all is working great!.