java client to connect remote hadoop machine error

Question

I am writing a java client program which can interact with remote hadoop cluster and prints all the jobs which are running.

my local machine is able to ping the remote machine where hadoop is running. i tried below code, and stuck up with parameters. where can i get these configuration parameter values

Configuration conf = new Configuration();
// this should be like defined in your mapred-site.xml
conf.set("mapred.job.tracker", "Hadoopmaster:54311"); 
// like defined in hdfs-site.xml
conf.set("fs.default.name", "hdfs://namenode.com:9000");

System.out.println("got configuration : "+conf);
InetSocketAddress jobtracker = new InetSocketAddress("jobtracker.mapredhost.myhost", 8021);
JobClient jobClient = new JobClient(jobtracker, conf);
JobStatus[] jobs = jobClient.jobsToComplete();

        for (int i = 0; i < jobs.length; i++) {
            JobStatus js = jobs[i];
            if (js.getRunState() == JobStatus.RUNNING) {
                JobID jobId = js.getJobID();
                System.out.println(jobId);
            }
        }
    }

map-red.xml

<property>
  <name>mapred.job.tracker</name>
  <value>Hadoopmaster:54311</value>
  <description>test </description>
</property>

core-site.xml

<property>
  <name>fs.default.name</name>
  <value>hdfs://Hadoopmaster:54310</value>
  <description>test.</description>
</property>

Tom Sebastian Tom Sebastian · Accepted Answer · 2014-02-14T07:09:57

In the provided sample the configuration properties did't match for client and cluster, fs.default.name To connect to remote hadoop cluster through java, I did the following code:

public static void main(String a[]) {
     UserGroupInformation ugi
     = UserGroupInformation.createRemoteUser("root");

     try {


        ugi.doAs(new PrivilegedExceptionAction<Void>() {

            public Void run() throws Exception {

                conf = new Configuration();
                    //fs.default.name should match the corresponding value 
                    // in your core-site.xml in hadoop cluster
                conf.set("fs.default.name","hdfs://hostname:9000");
                conf.set("hadoop.job.ugi", "root");
                    // in case you are running mapreduce job , need to set
                    // 'mapred.job.tracker' as you did
                    conf.set("mapred.job.tracker", "hostname:port"); 

                 // do your code here. 

                return null;
            }
        });

    } catch (Exception e) {
        e.printStackTrace();
    }

}

If you need default values please check:

http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml

and

http://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

java client to connect remote hadoop machine error

1 Answers