"Type mismatch in key from map: expected org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.LongWritable" -Every thing looks right

1
votes

I am trying to write simple map reduce program to find largest prime number using new API (0.20.2). This is how my Map and reduce class look like… 

public class PrimeNumberMap extends Mapper<LongWritable, Text, IntWritable, IntWritable> {

public void map (LongWritable key, Text Kvalue,Context context) throws IOException,InterruptedException
{
    Integer value = new Integer(Kvalue.toString());
    if(isNumberPrime(value))
    {
            context.write(new IntWritable(value), new IntWritable(new Integer(key.toString())));
    }
}

boolean isNumberPrime(Integer number)
{
    if (number == 1) return false;
     if (number == 2) return true;

     for (int counter =2; counter<(number/2);counter++)
     {
         if(number%counter ==0 )
             return false;
     }
            return true;

}
}
public class PrimeNumberReduce extends Reducer<IntWritable, IntWritable, IntWritable, IntWritable>  {

public void reduce ( IntWritable primeNo, Iterable<IntWritable> Values,Context context) throws IOException ,InterruptedException
{
    int maxValue = Integer.MIN_VALUE;
    for (IntWritable value : Values)
    {
        maxValue=  Math.max(maxValue, value.get());
    }
    //output.collect(primeNo, new IntWritable(maxValue));
    context.write(primeNo, new IntWritable(maxValue));  }

}  

 public static void main(String[] args)  throws IOException, InterruptedException, ClassNotFoundException{

    if (args.length ==0)
    {
        System.err.println(" Usage:\n\tPrimenumber <input Directory> <output Directory>");
        System.exit(-1);
    }
    Job job = new Job();
    job.setJarByClass(Main.class);

    job.setJobName("Prime");
    // Creating job configuration object

    FileInputFormat.addInputPath(job, new Path (args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.setMapOutputKeyClass(IntWritable.class);
    job.setMapOutputValueClass(IntWritable.class);

    job.setOutputKeyClass(IntWritable.class);
    job.setOutputValueClass(IntWritable.class);
    String star ="*********************************************";
    System.out.println(star+"\n Prime number computer \n"+star);
    System.out.println(" Application started ... keeping fingers crossed :/ ");
    System.exit(job.waitForCompletion(true)?0:1);

}

}

I am still getting error regarding mismatch of key for map

java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1034) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:595) at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:668) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109) at org.apache.hadoop.mapred.Child.main(Child.java:264) 2012-06-13 14:27:21,116 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

Can some one please suggest what is wrong. I have tried all hooks and crooks.

3
hadoop

3 Answers

8
votes

You've not configured the Mapper or reducer classes in your main block, so the default Mapper is being used - which is known as the identity mapper - each pair it receives as input is output (hence the LongWritable as the output key):

job.setMapperClass(PrimeNumberMap.class);
job.setReducerClass(PrimeNumberReduce.class);
0
votes

  1. The mapper should be defined as below,

    public class PrimeNumberMap extends Mapper<**IntWritable**, Text, IntWritable, IntWritable> {

    instead of 

    public class PrimeNumberMap extends Mapper<LongWritable, Text, IntWritable, IntWritable> {

  2. As it is mentioned in the comment before you should have the mapper and reducer defined.

    job.setMapperClass(PrimeNumberMap.class);
job.setReducerClass(PrimeNumberReduce.class);

Please refer to Hadoop Definitive guide 3rd edition, Chapter 2, Page 24

0
votes

I am a fresh hand in hadoop mapreduce program.

When mapping, I use IntWritable but I reduce the values in IntWritable format and convert the result to double before using DoubleWritable in context write.

It fails when running.

My method to handle the covert int in map to double in reduce is:

Mapper(LongWritable,Text,Text,DoubleWritable)
Reducer(Text,DoubleWritable,Text,DoubleWritable)
job.setOutputValueClass(DoubleWritable.Class)