running multiple MapReduce jobs in hadoop

Question

I want to run a chain of map reduce jobs, so the easiest solution seems to be jobcontroller. say I have two jobs, job1 and job2. and I want to run job2 after job1. Well, it faced some problems. after hours of debugging, I narrowed down the code to these lines:

JobConf jobConf1 = new JobConf();  
JobConf jobConf2 = new JobConf();  
System.out.println("*** Point 1");
Job job1 = new Job(jobConf1);  
System.out.println("*** Point 2");
Job job2 = new Job(jobConf2);
System.out.println("*** Point 3");

I keep getting this output when running the code:

*** Point 1    
10/12/06 17:19:30 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
*** Point 2    
10/12/06 17:19:30 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
*** Point 3

What I guess is that my problem is related somehow to "cannot initialize JMV ...." line. What is that? And how I can instantiate more than one job, in order to pass them to JobController.

When I added job1.waitForTheCompletion(true) before initializing the second job, it gave me this error:

    10/12/07 11:28:21 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/home/workspace/WikipediaSearch/__TEMP1
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
        at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
        at ch.ethz.nis.query.HadoopQuery.run(HadoopQuery.java:353)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at ch.ethz.nis.query.HadoopQuery.main(HadoopQuery.java:308)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

__Temp1 is the output folder of the first job, which I want to be the input for the second one. And even though I have this waitForCompletion line in my code, it's still complaining that this path doesn't exist.

narengi narengi · Accepted Answer · 2010-12-07T15:15:01

Wowww, after two days of debugging, it turns out that the problem is with hadoop internal directories names rule. Seemingly, for the input or output map-reduce directories, one cannot choose names starting with underline "_". That stupid! And the warnings and error were no help at all.

running multiple MapReduce jobs in hadoop

2 Answers