Type mismatch in value from map

Question

I am running this code using oozie workflow and getting type mismatch error:

 public static class mapClass extends Mapper<Object, Text, LongWritable, LongWritable> {
        public void map(Object, Text, Context..)
..
         context.write(<LongWritable type> , <LongWritable type> )
}

public static class reduceClass extends Reducer<LongWritable, LongWritable,LongWritable, LongWritable> {
        public void reduce(LongWritable, LongWritable, context)
..
    context.write(<LongWritable type>, <LongWritable type>)
{

}

java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.LongWritable, recieved org.apache.hadoop.io.Text

I'm using new-api in my workflow. The same code works fine without using oozie.

Any help would be appreciated. Thanks.

-----code sample ---

package org.apache.hadoop;
import java.io.IOException;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;


public class MapperLong extends Mapper<LongWritable, Text, LongWritable, LongWritable> {
    public final static int COL_ZERO = 0;
    public final static int COL_ONE = 1;
    public final static int COL_TWO = 2;
    public final static int COL_THREE = 3;

@Override
    public void map(LongWritable offset, Text line, Context context)
throws IOException, InterruptedException {
        String[] parts = (line.toString()).split(" ");

        LongWritable one = new LongWritable(Integer.parseInt(parts[COL_ONE]));
        LongWritable two = new LongWritable(Integer.parseInt(parts[COL_TWO]));
        context.write(one, two);

    }
}




package org.apache.hadoop;

import java.io.IOException;
import java.util.HashSet;
import java.util.Set;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.mapreduce.Reducer;


public class ReducerLong extends Reducer<LongWritable, LongWritable, LongWritable, LongWritable> {
@Override
    public void reduce(LongWritable colOneKey, Iterable<LongWritable> values,
        Context context) throws IOException, InterruptedException{
        Set<Integer> colTwo = new HashSet<Integer>();

        for (LongWritable val : values) {
            colTwo.add(Integer.valueOf((int)val.get()));
        }
        context.write(colOneKey, new LongWritable(colTwo.size()));
    }
}
}

java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.LongWritable, recieved org.apache.hadoop.io.Text at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574) at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264)

Input :
34 342 1 1
45 23 0 1
..
..

Note: I changed Object type to LongWritable which did not make any difference. The above exception is thrown while using the following property in workflow.xml. Without the following property, code executes producing output same as input prefixed with offset!

        <property>
            <name>mapred.output.key.class</name>
            <value>org.apache.hadoop.io.LongWritable</value>
        </property>
        <property>
            <name>mapred.output.value.class</name>
            <value>org.apache.hadoop.io.LongWritable</value>
        </property>

In general, don't forget that Java generics are not enforced at Runtime, so anybody can put any object in the map as they wish. So if something put in a Hadoop Text instance instead of a Hadoop LongWritable, you would get this message. More code, full stacktrace of exception would be helpful. — Maarten Bodewes
Agreed with owlstead. I'd feel better if the Mapper's first template item is not Object. — Donald Miner

hadoopQ hadoopQ · Accepted Answer · 2011-12-30T06:06:38

Ok, I figured out. The problem was in the oozie workflow I defined.

I had 
<name>mapreduce.mapper.class</name>
and 
<name>mapreduce.reducer.class</name>

instead of
<name>mapreduce.map.class</name>
and 
<name>mapreduce.reduce.class</name>

[refer https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases MR Api ] For some reason that could not catch my eyes :-( since I was using the modified version of non-api workflow!

Thank you all for your time.

Type mismatch in value from map

2 Answers