Mapred Child Error : AWS EMR Instance terminated

0
votes

When I was running EMR instances for medium type, I found in one of the step the EMR instance terminated. I used medium instances in AWS for 5 instances. It's capacity is 3.75 GB RAM. I gave heap size as 3000 MB for the attribute mapred.child.java.opts. I have set this using JobConf as gave parameter like conf.set("mapred.child.java.opts", "-Xmx3000m"). I had three steps to run for Amazon EMR program. In the final step, I got this problem, java.lang.Throwable: Child Error. What would be the reason job got failed. Also how much is the optimal heap size required for mappers in the child node.

1
javahadoopamazon-ec2heapamazon-emr
Can you give your full stack trace? Also there is no "optimal heap size", it totally depends on the nature of your job, the amount of data, and many other parameters, you should provide more details also on what you're trying to do exactly. - Charles Menguy
My stack trace: Will make it short. java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 137. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 137. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) - prasanna

1 Answers

0
votes

unless you are doing something special - 3000M is way too high a number. can you check how many map tasks hadoop is running per machine? if it's running more than one - it will blow away the 3.75GB ram easy (some has to be free for kernel anyway).

we provide hadoop as service (Qubole). by default here (and in my previous gigs) - we use 1024M as the default. you will want to set io.sort.mb well below that (say 400-500MB). only if the user is doing something special (like a large map-side join) - would there be reason to bump it up. if it had to be increased - we would have to reduce the max number of concurrent map tasks per machine (if it was more than one).