Cluster simulator for Hadoop MapReduce

1
votes

I am looking for a cluster simulator to test my Hadoop MapReduce Java applications (Driver, Mappers, Reducers, ...).

There is something that could simulate the HDFS, the execution of the tasks of each virtual nodes?

I am not clearly interested in performances, I have a Intel Core i7 and 16GB RAM maybe that's enough to simulate a small cluster.

1
mapreducebigdatavirtual-machinecluster-computingsimulator

1 Answers

1
votes

There are multiple ways to simulate your cluster.

  1. Hortonworks or cloudera docker image , which you can spawn and connect to it in your code.

  2. Run MR locally by setting conf.set("fs.default.name", "file:///"); conf.set("mapred.job.tracker", "local");

    See Integration test in : http://bytepadding.com/big-data/map-reduce/word-count-map-reduce/

  3. Use hbase-testing-util which spawns service of HDFS , HBASE as threads and cab be run loacally.

    http://bytepadding.com/big-data/hbase/hbase-readwrite-from-map-reduce/