We are trying to fetch the spark metrics using prometheus. we are using jmx exporter jmx_prometheus_javaagent-0.12.0.jar.
We are able to get the metrics for deploy-mode client but for cluster we are not able to get it.
For standalone command used is
./spark-submit --conf "spark.driver.extraJavaOptions=-javaagent:../jars/jmx_prometheus_javaagent-0.12.0.jar=8071:../conf/spark.yml" --class somecustomeclass --deploy-mode client --executor-memory 3g --driver-memory 3G ../jars/somejar.jar
For cluster we tried
./spark-submit --supervise --deploy-mode cluster --conf 'spark.driver.extraJavaOptions=-javaagent:jars/jmx_prometheus_javaagent-0.12.0.jar=8060:/conf/spark.yml spark.executor.extraJavaOptions=-javaagent:/jars/jmx_prometheus_javaagent-0.12.0.jar=8061:/conf/spark.yml spark.jars=/jars/jmx_prometheus_javaagent-0.12.0.jar ' --files /jars/jmx_prometheus_javaagent-0.12.0.jar --class org.apache.spark.examples.SparkTC --master spark://host1:7077,host2:7077 --executor-memory 1G --total-executor-cores 1 /spark/spark-2.4.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.0.jar 1
Are we missing anything in setup or command?
