How to point Hadoop Configuration class to specific directory

Question

The overloaded Configuration constructors include the following, none of which involve specifying the hadoop configuration directory.

    public Configuration() { /* compiled code */ }

    public Configuration(boolean loadDefaults) { /* compiled code */ }

    private Configuration(org.apache.hadoop.conf.Configuration other, boolean storeResource) { /* compiled code */ }

    public Configuration(org.apache.hadoop.conf.Configuration other) { /* compiled code */ }

So then, how to do so?

The use case is running a unit test on a machine running a pseudo-distributed hadoop cluster.

The unit test will by default try to access the local file system instead of the local HDFS. So the usefulness is to point the Configuration to a non-local set of core-*.xml files configured for pseudo-distributed mode in a specified $HADOOP_HOME/conf directory.

WestCoastProjects WestCoastProjects · Accepted Answer · 2014-05-11T07:34:56

A clumsy workaround: The Cxonfiguration class has a method to permit resources to be added after the fact" as follows:

addResource(InputStream in) 
      Add a configuration resource.

So a FileInputStream pointing to each of the *-site.xml's could be used.

How to point Hadoop Configuration class to specific directory

1 Answers