Spark accumulator not displayed in spark WebUI

12
votes

I'm using spark streaming. According to the Spark Programming Guide (see http://spark.apache.org/docs/latest/programming-guide.html#accumulators), named accumulators will be displayed in the WebUI as below: Accumulators in Spark WebUI Unfortunately, I cannot find this anywhere. I am registering the accumulators like this (Java):

LongAccumulator accumulator = new LongAccumulator();    
ssc.sparkContext.sc().register(accumulator, "my accumulator");

I am using Spark 2.0.0.

3
apache-spark
Accumulator of what type do you have? - Alexander Ponomarev
I'm having the same problem. I'm using a LongAccumulator and a custom one, and neither appear. Both are registered with a name using sc.register(). Were you able to figure it out? - Nate
Did you ever figure this out? I'm seeing the same issue in Scala. - Ross117

3 Answers

2
votes

I do not have a working streaming example but in non streaming example this UI could be found at the stages tab when choosing a specific stage. Also, I generally create the accumulator like this: 

val accum = sc.longAccumulator("My Accumulator")

The equivalent in for spark streaming would probably be to replace sc with ssc.SparkContext

1
votes

It worked for me. Below is my sample code

Accumulator<Integer> spansWritten = jsc.sparkContext().intAccumulator(0,"Spans_Written");
JavaDStream<Span> dStream = SourceFactory.getSource().createStream(jsc)
    .map( s -> {
      spansWritten.add(1);
      return s;
    });

However, when I tried to use them inside a Decoder while creating stream for kafka, it didn't show up in the UI.

Here is how it looks in the UI (select stages tab from the top, and click on one of the stage) screen shot

0
votes

Make sure to register the accumulator at your spark context object: 

LongAccumulator accumulator = new LongAccumulator();    
ssc.sparkContext().register(accumulator, "my accumulator");