Hadoop Distributed Cache throws IOException

Question

using hadoop 0.20.2 and trying to read a serialized map via distributed cache

facing a compilation error localFiles = DistributedCache.getLocalCacheFiles(job); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Unhandled exception type IOException

DRIVER Class snippet (/scratch/word_id.ser is a serialized file stored in local system)

    Job job = new Job(conf, "xml-read");
    DistributedCache.addCacheFile(new URI("/scratch/word_id.ser"),job);

MAPPER Class snippet

    public class MyParserMapper1 {

    public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, IntWritable, Text> {

    private FileSystem fs;
    private Path[] localFiles;
    HashMap  hash_temp;
    private ObjectInputStream oisc;

    @Override
    public void configure(JobConf job)   {

     localFiles = DistributedCache.getLocalCacheFiles(job);

    }

Chris White Chris White · Accepted Answer · 2012-04-18T10:17:52

Your IDE probably has some auto-fix rules for things like this, but anyway, you need to wrap the statement in a try / catch block:

@Override
public void configure(JobConf job)   {
  try {
    localFiles = DistributedCache.getLocalCacheFiles(job);
  } catch (IOException ioe) {
    throw new RuntimException(ioe);
  }
}

If you can handle the exception all the better (i.e. if you can still run your mapper without this file), but otherwise just wrap in an unchecked exception like RuntimeException

Hadoop Distributed Cache throws IOException

1 Answers