hadoop map reduce job no output

Question

I'm write MapReduce job in Netbeans and generate (also in NB) a jar file. When I try to execute this job in hadoop (version 1.2.1) I execute this command:

$ hadoop jar job.jar org.job.mainClass /home/user/in.txt /home/user/outdir

This command not show any errors but not create outdir, outfiles, ...

This is my job code:

Mapper

public class Mapper extends MapReduceBase implements org.apache.hadoop.mapred.Mapper<LongWritable, Text, Text, IntWritable> {

            private final IntWritable one = new IntWritable(1);
            private Text company = new Text("");


            @Override
            public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
                company.set(value.toString());
                output.collect(value, one);

            }

        }

Reducer

public class Reducer extends MapReduceBase implements org.apache.hadoop.mapred.Reducer<Text, IntWritable, Text, IntWritable> {

    @Override
    public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {

        int sum = 0;
        while (values.hasNext()){
            sum++;
            values.next();
        }

        output.collect(key, new IntWritable(sum));
    }
}

Main

 public static void main(String[] args) {

    JobConf configuration = new JobConf(CdrMR.class);
    configuration.setJobName("Dedupe companies");
    configuration.setOutputKeyClass(Text.class);
    configuration.setOutputValueClass(IntWritable.class);
    configuration.setMapperClass(Mapper.class);
    configuration.setReducerClass(Reducer.class);
    configuration.setInputFormat(TextInputFormat.class);
    configuration.setOutputFormat(TextOutputFormat.class);
    FileInputFormat.setInputPaths(configuration, new Path(args[0]));
    FileOutputFormat.setOutputPath(configuration, new Path(args[1]));

}

The format of input file is as follows:

name1
name2
name3
...

Also say I'm executing hadoop in virtual machine (Ubuntu 12.04) without root privileges. Are Hadoop executing the job and stored outfile in different dir?

from which user you are running hadoop and where are you storing the output? are they both same users? — Y.Prithvi
add this in last line of main method System.exit(configuration.waitForCompletion(true) ? 0 : 1); — Y.Prithvi

USB USB · Accepted Answer · 2014-09-11T05:55:58

The correct hadoop command is

hadoop jar myjar packagename.DriverClass input output

CASE 1

MapReduceProject
    |
    |__ src
         |
         |__ package1
            - Driver
            - Mapper
            - Reducer

Then You can just use

hadoop jar myjar input output

CASE 2

MapReduceProject
    |
    |__ src
         |
         |__ package1
         |  - Driver1
         |  - Mapper1
         |  - Reducer1
         |
         |__ package2
            - Driver2
            - Mapper2
            - Reducer2

For this case you must specify driver class along with your hadoop command.

hadoop jar myjar packagename.DriverClass input output

hadoop map reduce job no output

3 Answers