Hadoop mapreduce custom writable static context

Question

I'm working on an university homework and we have to use hadoop mapreduce for it. I'm trying to create a new custom writable as I want to output key-value pairs as (key, (doc_name, 1)).

public class Detector {

    private static final Path TEMP_PATH = new Path("temp");
    private static final String LENGTH = "gramLength";
    private static final String THRESHOLD = "threshold";


    public class Custom implements Writable {

        private Text document;
        private IntWritable count;

        public Custom(){
            setDocument("");
            setCount(0);
        }

        public Custom(String document, int count) {
            setDocument(document);
            setCount(count);
        }

        @Override
        public void readFields(DataInput in) throws IOException {
            // TODO Auto-generated method stub
            document.readFields(in);
            count.readFields(in);
        }

        @Override
        public void write(DataOutput out) throws IOException {
            document.write(out);
            count.write(out);
        }

        public int getCount() {
            return count.get();
        }

        public void setCount(int count) {
            this.count = new IntWritable(count);
        }

        public String getDocument() {
            return document.toString();
        }

        public void setDocument(String document) {
            this.document = new Text(document);
        }

    }

    public static class NGramMapper extends Mapper<Text, Text, Text, Text> {
        private int gramLength;
        private Pattern space_pattern=Pattern.compile("[ ]");
        private StringBuilder gramBuilder= new StringBuilder();

        @Override
        protected void setup(Context context) throws IOException,      InterruptedException{
            gramLength=context.getConfiguration().getInt(LENGTH, 0);
        }

        public void map(Text key, Text value, Context context) throws IOException, InterruptedException {
            String[] tokens=space_pattern.split(value.toString());
            for(int i=0;i<tokens.length;i++){
                gramBuilder.setLength(0);
                if(i+gramLength<=tokens.length){
                    for(int j=i;j<i+gramLength;j++){
                        gramBuilder.append(tokens[j]);
                        gramBuilder.append(" ");
                    }
                    context.write(new Text(gramBuilder.toString()), key);
                }
            }
        }
    }


    public static class OutputReducer extends Reducer<Text, Text, Text, Custom> {

        public void reduce(Text key, Iterable<Text> values, Context context)
                throws IOException, InterruptedException {
            for (Text val : values) {
                context.write(key,new Custom(val.toString(),1));
            }
        }
    }

    public static void main(String[] args) throws Exception {

        Configuration conf = new Configuration();
        FileSystem fs = FileSystem.get(conf);
        conf.setInt(LENGTH, Integer.parseInt(args[0]));
        conf.setInt(THRESHOLD, Integer.parseInt(args[1]));

        // Setup first MapReduce phase
        Job job1 = Job.getInstance(conf, "WordOrder-first");
        job1.setJarByClass(Detector.class);
        job1.setMapperClass(NGramMapper.class);
        job1.setReducerClass(OutputReducer.class);
        job1.setMapOutputKeyClass(Text.class);
        job1.setMapOutputValueClass(Text.class);
        job1.setOutputKeyClass(Text.class);
        job1.setOutputValueClass(Custom.class);
        job1.setInputFormatClass(WholeFileInputFormat.class);
        FileInputFormat.addInputPath(job1, new Path(args[2]));
        FileOutputFormat.setOutputPath(job1, new Path(args[3]));

        boolean status1 = job1.waitForCompletion(true);
        if (!status1) {
            System.exit(1);
        }
    }
}

When I compile the code to a class file i get this error:

Detector.java:147: error: non-static variable this cannot be referenced from a static context
context.write(key,new Custom(val.toString(),1));

I followed differents tutorials about custom writable and my solution is the same as the others. Any suggestion?

Writing a custom writable is overkill. Try using Avro: avro.apache.org — simleo

x80486 x80486 · Accepted Answer · 2016-01-08T14:24:40

Static fields and methods are shared with all instances. They are for values which are specific to the class and not a specific instance. Stay out of them as much as possible.

To solve your problem, you need to instantiate an instance (create an object) of your class so the run-time can reserve memory for the instance; or change the part you are accessing it to have static access (not recommended!).

The keyword this is for referencing something that's indeed an instance (hence the this thing) and not something that's static, which in that case should be referenced by the class name instead. You are using it in a static context which is not allowed.

Hadoop mapreduce custom writable static context

1 Answers