Typesafe Config in Spark

Question

I've defined a default configuration in my Spark application which is tucked in src/main/resources/reference.conf. I use ConfigFactory.load() to obtain the configuration.

When I run the application with spark-submit it picks up these defaults. However, when I only want to override a few of the configurations available in reference.conf and provide application.conf, it does not seem to pick up these overrides. From the documentation I thought that application.conf is merged with reference.conf when calling load(), so that it's not necessary to re-define everything in application.conf.

My reference.conf looks like this:

hdfs {
  rootDir: "/foo"
  dataDir: "hdfs://"${hdfs.rootDir}"/bar"
}

db {
  driver: "com.mysql.jdbc.Driver"
  ...
}

...

What I'd now like to do is have an application.conf with, say, only a custom hdfs section because the rest is the same.

I run my Spark app by supplying application.conf in both the --files parameter, in --driver-class-path, and --conf spark.executor.extraClassPath. This may be overkill but it works when I create a copy of reference.conf and change a few of the fields.

What am I missing?