PySpark Logging

0
votes

I'm trying to view the spark logs in GCP stack driver. Below is the screenshot of the stack driver.

enter image description here

As per the above screenshot Spark INFO's are mapping to ERROR in stack driver. Not sure why?

I'm using log4j for spark custom logging. Below are the log properties.

# Root logger option
log4j.rootLogger=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p    %c{1}:%L - %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark-project.jetty=INFO
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

Any suggestions to map spark log(INFO) to stack driver log(INFO)?

1
apache-sparkgoogle-cloud-platformpysparklog4jstackdriver

1 Answers

0
votes

Stackdriver reads some messages as the severity, in this case you have

log4j.appender.console.target=System.err

Try add a "Severity" field to your logs with the proper value and, this way, Stackdriver would read it as you specify, check the values here. Also you can check the correct format at the beginning of the page.