installing hadoop and writing map reduce program

0
votes

for a subject I have got this assignment.

To make a hadoop cluster and write map reduce programs.

I have a laptop with 4GB ram and i3 processor, I downloaded vmware image from cloudera website. But the pre configured virtual machine itself takes 4GB RAM.

The assignment text says

install the Hadoop Distribution of Cloudera (http://www.cloudera.com/hadoop/) in Pseudo-Distributed Mode or use the VMWare Image provided by Cloudera to familiarize yourself with Hadoop, especially with the distributed file system HDFS and the implementation of MapReduce programs in Java."

I downloaded vmware image from cloudera website, but the pre configured virtual machine itself takes 4GB RAM.

I tried reducing the size of Virtual machine memory from 4GB to 1 GB but it was not good; I mean I could not run the cloudera virtual machine.

I have a lot of mapreduce and java programs which the assignment says me to do. I am not able to understand any of them. Like

  • doing a "grep" on multiple machines.
  • Counting word frequency on files spread across multiple machines in hadoop cluster etc etc.

I want to know how do I setup hadoop so that it runs on windows8.1 machine so that I can run these programs

1
javahadoopmapreduce
Asking for books, tutorials, and other off site resources is considered off-topic for StackOverflow. "Hadoop : Definitive Guide" is considered the "Hadoop bible", though - OneCricketeer

1 Answers

0
votes

Cloudera VM requires 6-8GB to run correctly.

When I took a Hadoop course in university, it was required for us to buy more RAM for all computers with less than 8GB, and we had i5's but the VM is still really slow.

Even just installing Hadoop and running the services alone outside of a VM will require a minimum of 4GB, by default. That's not including your OS and other services (your browser and OS are probably already taking 1GB each just by themselves).

As far as actually installing Hadoop on Windows, I wouldn't recommend it, but rough steps are

  1. Install Java. Add JAVA_HOME as environment variable
  2. Install and run an SSH server on your windows machine. Make sure you can connect to localhost:22 using PuTTy, for example
  3. Then download and configure Hadoop using Apache site, not random tutorials elsewhere that could be out-of-date. Start at Single Node, then configure Pseudo-distributed. As soon as you extract Hadoop download, add HADOOP_PREFIX and HADOOP_CONF_DIR=%HADOOP_PREFIX%/conf as two environment variables
  • doing a "grep" on multiple machines
  • Counting word frequency on files

Both of these are the examples given in the documentation. Not sure you are required to actually write that code.

FWIW, you don't actually need a running Hadoop cluster to run MapReduce. The default Hadoop configurations will read from your single, local filesystem. Besides, your VM is a single machine anyway, so the requirement of "running on multiple machines" doesn't make much sense.