NullPointer Exception in map task in mapreduce

Question

I am running a program to extract only english tweets. However after the map task is completed and reduce is about to begin, it throws a null pointer exception and I couldn't figure out why. It was working fine on one input set but not on another.

Mapper task:

public void map(Object Key, Text value, Context context) throws IOException, InterruptedException {
    String lang = null;
    String eng = "en";

    try {
        twitter4j.Status s = DataObjectFactory.createStatus(value.toString());
        User user = s.getUser();
        lang = user.getLang();
        System.out.println(lang1);
        if (lang.equals(eng)) {
            word.set(value.toString());
            context.write(word, NullWritable.get());
        }
    } catch (TwitterException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    // }
}

Reducer task:

public static class TReducer extends Reducer<Text, NullWritable, Text, NullWritable> {
    private Text word = new Text();
    private IntWritable result = new IntWritable();

    public void reduce(Text key, NullWritable values, Context context) throws IOException, InterruptedException {
        word.set(key);
        context.write(word, NullWritable.get());
    }
}

Stack trace:

2015-07-29 11:37:16,245 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1440)) - Starting flush of map output
2015-07-29 11:37:16,245 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1459)) - Spilling map output
2015-07-29 11:37:16,245 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1460)) - bufstart = 0; bufend = 1814038; bufvoid = 104857600
2015-07-29 11:37:16,246 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1462)) - kvstart = 26214396(104857584); kvend = 26211056(104844224); length = 3341/6553600
2015-07-29 11:37:16,256 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1648)) - Finished spill 0
2015-07-29 11:37:16,270 INFO  [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:run(397)) - Map task executor complete.
2015-07-29 11:37:16,367 WARN  [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:run(482)) - job_local1233871598_0001
java.lang.Exception: java.lang.NullPointerException
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:403)
Caused by: java.lang.NullPointerException
    *at lang$TweetMapper.map(lang.java:180)* --> points to line lang=user.getlang()
    at lang$TweetMapper.map(lang.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2015-07-29 11:37:17,334 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1358)) - Job job_local1233871598_0001 failed with state FAILED due to: NA
2015-07-29 11:37:17,548 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1363)) - Counters: 26
    File System Counters
        FILE: Number of bytes read=6515040861457
        FILE: Number of bytes written=21228998775083
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=25094959861760
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=376376
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=611
    Map-Reduce Framework
        Map input records=27447674
        Map output records=16257677
        Map output bytes=47816627467
        Map output materialized bytes=47881661841
        Input split bytes=69043
        Combine input records=16257677
        Combine output records=16257677
        Spilled Records=23671923
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=133753
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
        Total committed heap usage (bytes)=2818542403584
    File Input Format Counters 
        Bytes Read=82009661440

Could someone tell me what is wrong here? IS it because of input or something else?

Naman Gala Naman Gala · Accepted Answer · 2015-07-29T11:19:15

Make sure you add null check before accessing user object. And find out why user object is null if it should not be null.

if (user != null) {
    lang=user.getLang();
    //    ^---------------- user is null.
    ...
}

NullPointer Exception in map task in mapreduce

1 Answers