Information on Nutch , Hadoop , Solr, MapReduce and Mahout

0
votes

PS: Correct me if I am wrong in any line

I am building a search engine with Nutch and Solr.
I know by using Solr, I can enhance the efficiency of Searching- let Nutch do the crawling alone of the entire web.
I also know that Hadoop is used to handle petabytes of data by forming clusters and MapReduce.

Now , What i want to know is that
1) Since,I'll be running these open source softwares on only 1 machine,ie, my laptop on localhost... How would Hadoop be beneficial in my case as it forms clusters? How would clusters be formed on only 1 machine??

2) What would be the importance of MapReduce in my case?

3) How would MAHOUT,CASSANDRA and HBASE effect my engine???

Any help on this aspect is very much appreciated.Apologize me if I asked a noob question!!
Thanks
Regards

1
solrhadoopmapreducenutchmahout
Please do some ground work before asking a question !!! - Praveen Sripati

1 Answers

5
votes

1) Since,I'll be running these open source software on only 1 machine,ie, my laptop on localhost... How would Hadoop be beneficial in my case as it forms clusters?

Hadoop was created to process large scale data. Hadoop is a distributed application. It is not going to provide you benefits on a single machine.

How would clusters be formed on only 1 machine??

Install Hadoop in pseudo cluster mode

What would be the importance of MapReduce in my case?

Again, if you want to process pages fetched by a crawler on the scale of 1000s of gigabyte. Map-Reduce is useful in processing such large data

How would MAHOUT,CASSANDRA and HBASE effect my engine???

They are different tools for different needs.

Mahout is machine learning algorithms adapted for running as map-reduce tasks on Hadoop or local files. Do you want to learn languages like Google Translate, you can use it.

HBase is a no-sql database that provides more real time data processing over ad hoc analysis for which map-reduce is more useful.

I would suggest that you go back to your problem statement, design with as little tools as required and when you hit the notes, you will understand when some of these tools could be useful.