Spring Dataflow and Yarn: How to set properties properly?

Question

How can one change the default appdeployappmaster properties ?

When I'm trying to deploy an application through Spring DataFlow YARN. I registered my app, created a stream, and click the "deploy" button. When doing so, I get the following error :

[XNIO-2 task-2] WARN o.s.c.d.s.c.StreamDeploymentController - Exception when deploying the app StreamAppDefinition [streamName=histo, name=my-app, registeredAppName=my-app, properties={spring.cloud.stream.bindings.input.destination=log, spring.cloud.stream.bindings.input.group=histo}]: java.util.concurrent.ExecutionException: org.springframework.yarn.YarnSystemException: Invalid host name: local host is: (unknown); destination host is: "null":8032; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost; nested exception is java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: "null":8032; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost

As you can see, the deployer is unable to find the "Resource Manager" URI, Although it is well found when the Spring DataFlow Server starts. So I only get the problem at the deployment time.

Which property should I set to fix this issue, and where would I do that ?

EDIT 1: Following Janne Valkealahti's answer, I added the following properties in /dataflow/apps/stream/app/servers.yml, relaunched the server, and tried to re-deploy my stream.

spring:
  cloud:
    dataflow:
      yarn:
        version: 0.0.1-SNAPSHOT
    deployer:
      yarn:
        version: 1.0.2.RELEASE
    stream:
      kafka:
        binder:
          brokers: kafka.my-domain.com:9092
          zkNodes: zookeeper.my-domain.com:2181/node
  # Configured for Hadoop single-node running on localhost. Replace with property values reflecting your
  # actual Hadoop cluster when running in a distributed environment.
  hadoop:
    fsUri: hdfs://mapr.my-domain.com/referentiel/ca_category_2014/
    resourceManagerHost: mapr.my-domain.com
    resourceManagerPort: 8032
    resourceManagerSchedulerAddress: mapr.my-domain.com:8030
  session:
    store-type: none

I still get the exact same message.

PS: I'm not using Ambari, I'd like to understand how it works manually first.

EDIT 2: I solved the problem adding the -Dspring.config.location VM arg on the DataFlow Server. The given configuration is passed to the deployer, and the application is effectively deployed.

I'll write an answer for it.

Janne Valkealahti Janne Valkealahti · Accepted Answer · 2016-12-02T18:15:46

You didn't tell if your installation was based on ambari or normal manual YARN install so I assume it was a latter(manual).

I think a problem is that in distribution you use the config/servers.yml has a wrong setting for resourceManagerHost as it defaults to localhost. This file is distribute only once into hdfs when streams are launched. If you have changed it after you redeploy/create stream, app in hdfs directory will not get updated. On default this file in hdfs is /dataflow/apps/stream/app/servers.yml.

This error makes sense as also dataflow yarn server controlling whole stuff also needs access to yarn resource manager to submit apps. Settings for server also comes from a same servers.yml file.

Spring Dataflow and Yarn: How to set properties properly?

2 Answers