Run Hadoop Java code without Hadoop command

Question

public void readFile(String file) throws IOException {
    Cofiguration conf = new Configuration();
    conf.addResource(new Path("/usr/local/hadoop-2.7.3/etc/hadoop/core-site.xml"))
    conf.addResource(new Path("/usr/local/hadoop-2.7.3/etc/hadoop/hdfs-site.xml"))
    conf.addResource(new Path("/usr/local/hadoop-2.7.3/etc/hadoop/mapred-site.xml"))
}
FileSystem fileSystem = FileSystem.get(conf);
System.out.println("DefaultFS:      " + cong.get("fs.defaultFS"));
System.out.println("Home directory: " + fileSystem.getHomeDirectory());

Path path = new Path(file);
if(!fileSystem.exists(path)) {
    System.out.println("File " + file + " does not exists");
    return;
}

I am very new to Hadoop and I am wondering if it is possible to execute this Hadoop Java Client code using "java -jar".

My code works using the "hadoop jar" command. However, when I try to execute this code using "java -jar" instead of "hadoop jar", it can't locate the file in HDFS and the method getHomeDirectory() returns a local path that doesn't exist.

Is my configuration files not added correctly? Why does the code only work when executed under Hadoop command?

Karun Mathew Karun Mathew · Accepted Answer · 2017-11-08T12:57:30

Instead of passing a Path object, pass the file path as string

    conf.addResource("/usr/local/hadoop-2.7.3/etc/hadoop/core-site.xml");
    conf.addResource("/usr/local/hadoop-2.7.3/etc/hadoop/hdfs-site.xml");
    conf.addResource("/usr/local/hadoop-2.7.3/etc/hadoop/mapred-site.xml");

Or else you could add these files to classpath and try.

    conf.addResource("core-site.xml");
    conf.addResource("hdfs-site.xml");
    conf.addResource("mapred-site.xml");

Run Hadoop Java code without Hadoop command

1 Answers