log4j.properties file not found on classpath or ignored

Question

I want to log in maprDB a spark job with log4j. I have written a custom appender, and here my log4j.properties :

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.Target=System.out log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

log4j.appender.MapRDB=com.datalake.maprdblogger.Appender

log4j.logger.testest=WARN, MapRDB

Put on src/main/resources directory

This is my main method :

object App {
    val log: Logger = org.apache.log4j.LogManager.getLogger(getClass.getName)

    def main(args: Array[String]): Unit = {
        // custom appender
        LogHelper.fillLoggerContext("dev", "test", "test", "testest", "")
        log.error("bad record.")
    }
}

When I run my spark-submit without any configuration, nothing happens. It is like my log4j.properties wasn't here.

If I deploy my log4j.properties file manually and add options :

--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/PATH_TO_FILE/log4j.properties

--conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/PATH_TO_FILE/log4j.properties

It works well. Why it doesn't work without theese options ?

What build tool do you use? Shouldn't the path be src/main/resources instead? — Jacek Laskowski
I also think log4j.appender.stdout.Target (uppercase) should be log4j.appender.stdout.target (lowercase). — Jacek Laskowski
The log4j.properties works well if I set the path explicitly. The problem is spark and his log4j.properties which is selected in first. — Franck Cussac
Use -Dlog4j.debug and see where log4j searches for the properties. — Jacek Laskowski

Sarfaraz Shaikh Sarfaraz Shaikh · Accepted Answer · 2019-07-25T07:15:05

The "spark.driver.extraJavaOptions" :

Default value is: (none)

A string of extra JVM options to pass to the driver. For instance, GC settings or other logging. Note that it is illegal to set maximum heap size (-Xmx) settings with this option. Maximum heap size settings can be set with spark.driver.memory in the cluster mode and through the --driver-memory command-line option in the client mode.

Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file.

[Refer to this link for more details: https://spark.apache.org/docs/latest/configuration.html]

log4j.properties file not found on classpath or ignored

1 Answers