Apache spark output messages are prepended by [error]

Question

This word count works as expected :

System.setProperty("hadoop.home.dir", "H:\\winutils");

val sparkConf = new SparkConf().setAppName("GroupBy Test").setMaster("local[1]")
val sc = new SparkContext(sparkConf)

def main(args: Array[String]) {
  val text_file = sc.textFile("h:\\data\\small.txt")

  val counts = text_file.flatMap(line => line.split(" "))
      .map(word => (word, 1))
      .reduceByKey(_ + _)

  counts.foreach(println);
}

All output messages are prepended by [error] example :

[error] 16/03/17 12:13:58 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on por
[error] 16/03/17 12:13:58 INFO NettyBlockTransferService: Server created on 55715
[error] 16/03/17 12:13:58 INFO BlockManagerMaster: Trying to register BlockManager
[error] 16/03/17 12:13:58 INFO BlockManagerMasterEndpoint: Registering block manager localhost:55715 with 1140.4 MB RAM, BlockManage
[error] 16/03/17 12:13:58 INFO BlockManagerMaster: Registered BlockManager

I can prevent these error messaged being displayed using :

import org.apache.log4j.Logger
import org.apache.log4j.Level

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

But this does not fix the issue.

[error] should not be displayed as these are not error messages but are info :

[error] 16/03/17 12:13:58 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on por
[error] 16/03/17 12:13:58 INFO NettyBlockTransferService: Server created on 55715
[error] 16/03/17 12:13:58 INFO BlockManagerMaster: Trying to register BlockManager
[error] 16/03/17 12:13:58 INFO BlockManagerMasterEndpoint: Registering block manager localhost:55715 with 1140.4 MB RAM, BlockManage
[error] 16/03/17 12:13:58 INFO BlockManagerMaster: Registered BlockManager

Update :

Why are [error] messages being displayed as they are not errors ?

setting Logger.getLogger("org").setLevel(Level.OFF) Logger.getLogger("akka").setLevel(Level.OFF) will only make the error messages not displayed. it will not deal with the problem but 'kill the messanger' did you try setLevel(Level.Warn) ?? — Zahiro Mor
@Zahiro Mor it does not deal with the problem as I have said "But this does not fix the issue." my question is why are [error] messages being displayed as they are not errors ? — blue-sky

Mateusz Dymczyk Mateusz Dymczyk · Accepted Answer · 2016-03-17T13:43:17

Those are not Spark labels but sbt ones. In the default log4j config file of Spark you can find:

log4j.appender.console.target=System.err

So by default it will print to stderr in the console.

You probably are setting fork to true in your run config somewhere. When doing so everything that is printed to stderr in sbt is prepended with [error].

You should be able to control it with the OutputStrategy.

Apache spark output messages are prepended by [error]

1 Answers