Java Heap Space Error in running mahout item similarity job on Amazon EMR

1
votes

I am trying to run a mahout item similarity job on a input consists of ~250 Million Pairs(row) in a Amazon EMR Cluster(m3.2xLarge,10 core nodes).I am facing Java Heap Size error while running the similarity job.

Things i have tried to solve this issue.

  1. Increase the heap size of name nodes by defining them in bootstrap action.Like this -
    --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-daemons --args --namenode-heap-size=8192

  2. Use memory intensive bootstrap recommended by AWS (s3://elasticmapreduce/bootstrap-actions/configurations/latest/memory-intensive)

  3. Set MAHOUT_HEAPSIZE manually.

The problem isn't solved.Is there any way to solve it?

1
javamahoutamazon-emrmahout-recommender
Is there any possible way for you to give the 250M inputs in different segments? Do you get the heap error immediately after passing the inputs or is it occurring after some particular period? – Payam
Not actually.It runs for some period of time then get heap space error. – Shihab Rahman
Alright, the problem is You really have not enough memory to run this huge bulk of data (meaning that you need to find a better PC) OR you have a memory leak. Are you using an IDE to run this? If yes just profile your java application and you can see the behavior of your heap. It's either getting bigger and bigger till it reaches the limit (option 1) or it's varying through the application with a weird manner (GC is not working properly). – Payam
Netbeans Profiler, Eclipse Profiler, Jprofiler are tools which can help you to deal with heap issues. – Payam
I am running this on an Amazon EMR cluster.So I don't get how an IDE profiler would help. – Shihab Rahman

1 Answers

2
votes

It is rare to need to change the default namenode daemon heap size on EMR and more likely the container/task jvm that needs tuning (http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/TaskConfiguration_H2.html).

Before making further changes you will need to determine what process is actually suffering from the heap error and then tune accordingly.

If it does turn out to be container/task jvm heap then it can be adjusted with configure-hadoop (http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-bootstrap.html#PredefinedbootstrapActions_ConfigureHadoop).

Also drop the memory-intensive bootstrap action, it is only for AMI 1.x.