Accessing HDFS files from eclipse

0
votes

I am a newbie to Hadoop. I am facing issues with accessing the files in my HDFS system from my java code in eclipse. Although my fs.default.name is set to hdfs://localhost:portno in core-site.xml it gives me the URI as file:// instead of hdfs:// I tried couple of other things like setting my input path as below : FileInputFormat.setInputPaths(conf, new Path("hdfs://localhost:9021/user/training/shakespeare/poems"));

but it throws the timeout error : 11/10/17 15:31:31 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9021. Already tried 0 time(s).

Please guide me how to resolve this issue, i am badly stuck cuz of this. Any help would really be appreciated.

Thanks

2
hadoop
Have you tried to connect from the command line? I.e. something like this: telnet 127.0.0.1:9021 If you still get a confection refused then the problem most likely lies with a local firewall. - Niels Basjes

2 Answers

3
votes

You need to make sure that {$HADOOP_HOME}/conf folder containing core-site.xml, hdfs-site.xml and mapred-site.xml is in the Java CLASSPATH while you run the program from Eclipse.

To add this folder to CLASSPATH right-click the folder->properties->Java Build Path->Add External Class Folder.

1
votes

If you do not want to specify *-site.xml files, you can simply configure the hadoop client directly in the code:

Configuration conf = new Configuration();
conf.set("fs.defaultFS","hdfs://namenode:8020");
FileSystem fs = FileSystem.get(conf);

If you use Maven in your project, then you can add the hadoop client into your pom.xml in the following way (assuming you are choosing the cloudera distribution):

<repositories>
    <repository>
        <id>cloudera</id>
        <url>https://repository.cloudera.com/content/repositories/releases/</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.6.0-cdh5.5.0</version>
    </dependency>
</dependencies>