Hadoop MapReduce DistributedCache usage

Question

I'm trying to reproduce the Bloom Filtering example of MapReduce Design Pattern book.
In the following, I will show only the code of interest:

public static class BloomFilteringMapper extends Mapper<Object, Text, Text, NullWritable>
{
    private BloomFilter filter = new BloomFilter();

    protected void setup( Context context ) throws IOException
    {
        URI[] files = DistributedCache.getCacheFiles( context.getConfiguration() );
        String path = files[0].getPath();
        System.out.println( "Reading Bloom Filter from: " + path );
        DataInputStream strm = new DataInputStream( new FileInputStream( path ) );
        filter.readFields( strm );
        strm.close();
    }
    //...
}
public static void main( String[] args ) throws Exception
{
    Job job = new Job( new Configuration(), "description" );
    URI uri = new URI("hdfs://localhost:9000/user/draxent/comment.bloomfilter");
    DistributedCache.addCacheFile( uri, job.getConfiguration() );
    //...
}

When I try to execute it, I receive the following error:
java.io.FileNotFoundException: /user/draxent/comment.bloomfilter

But executing the command:

bin/hadoop fs -ls

I can see the file:

-rw-r--r--   1 draxent supergroup        405 2015-11-25 17:12 /user/draxent/comment.bloomfilter

So I am quite sure the problem is on the line:

URI uri = new URI("hdfs://localhost:9000/user/draxent/comment.bloomfilter");

But I have tried several different configuration, like:
"hdfs://user/draxent/comment.bloomfilter"
"/user/draxent/comment.bloomfilter"
"comment.bloomfilter"

And no one works.

I have tried to look up at the cfeduke implementation, but I was no able to solve my problem.

Answer comments:

• ravindra: URI files[0] contains the string element passed in the main;
• Manjunath Ballur: yea, you are right. But since the file exists (you can see it from bin/hadoop fs -ls) this means that is a problem of the string path passed to FileInputStream. But I'm passing the string to that like always. I checked, the path value is: comment.bloomfilter... so it has to be right.

Answer 1

The Distributed Cache API has been deprecated.

You can extend the same functionality using the new API.Check the documentation here: http://hadoop.apache.org/docs/stable2/api/org/apache/hadoop/mapreduce/Job.html

In the driver code:-

Job job = new Job(); ... job.addCacheFile(new Path(filename).toUri());

In the mapper setup method :-

Path[] localPaths = context.getLocalCacheFiles();

Answer 2

The following should work: remove the line with URI uri = new URI(... and change the next line to:

DistributedCache.addCacheFile(new Path("/user/draxent/comment.bloomfilter").toUri(), job.getConfiguration());