Launching Spark job with Oozie fails (Error MetricsSystem)

0
votes

I have a spark jar that I launch with spark-submit and it works fine (reading files, generate RDD, storing in hdfs). However, when I tried to launch the same jar within an Oozie job (oozie:spark-action) the spark job fails.

When I looked the logs, the first error to shows up is :

Error MetricsSystem: Sink class org.apache.spark.metrics.sink.MetricsServlet cannot be instantiated.

Furthermore, when I started playing with the spark script, I found out that the problem has to do with saveAsText funtion. When I lunch the same spark job without writing to HDFS the whole workflow works fine.

Any suggestions ?

1
scalaapache-sparkooziespark-submit
which oozie action are you using? - Rahul Sharma
I am using a <spark xmlns="uri:oozie:spark-action:0.1> - OUMOUSS_ELMEHDI

1 Answers

0
votes

The problem was in the side of the cluste where i am executing oozie jobs.

I needed to explicitely add arguments in the job workflow, simply because they weren't taken into consideration:

<spark-opts>--queue HQ_IBNF --conf "spark.executor.extraJavaOptions=-Djava.library.path=/opt/application/Hadoop/current/lib/native"</spark-opts>