How to configure executors with custom StatsD Spark metrics sink

0
votes

How do I sink Spark Streaming metrics to this StatsD sink for executors?

Similar to other reported issues (sink class not found, sink class in executor), I can get driver metrics, but executors throw ClassNotFoundException with my setup:

  • StatsD sink class is compiled with my Spark-Streaming app (my.jar)

  • spark-submit is run with:

    • --files ./my.jar (to pull jar containing sink into executor)
    • --conf "spark.executor.extraClassPath=my.jar"

  • Spark Conf is configured in the driver with:

    val conf = new SparkConf()

conf.set("spark.metrics.conf.*.sink.statsd.class",
         "org.apache.spark.metrics.sink.StatsDSink")
    .set("spark.metrics.conf.*.sink.statsd.host", conf.get("host"))
    .set("spark.metrics.conf.*.sink.statsd.port", "8125")
2
scalaapache-sparkmetricssink

2 Answers

2
votes

Looks you hit the bug https://issues.apache.org/jira/browse/SPARK-18115. I hit it too and googled your question :(

1
votes

Copy your jar files to the $SPARK_HOME/jars folder.