How to skip failed map tasks in hadoop streaming

3
votes

I am running a hadoop streaming mapreduce job which has 26895 map tasks in total. However, one task that deals a certain input always fails. So I set mapreduce.map.failures.maxpercent=1 and want to skip failed tasks, but the job was still not successful. 

Kind % Complete  Num Tasks  Pending Running Complete    Killed  Failed/Killed Task Attempts

map     100.00%   26895     0       0       26894       1       8 / 44

reduce  100.00%       1     0       0       0           1       0 / 1

How can I do to skip this?

1
hadoopmapreducehadoop-streaming

1 Answers

2
votes

There is a configuration available for the same. Specify the mapred.max.map.failures.percent and mapred.max.reduce.failures.percent in the mapred-site.xml to specify the failure threshold. Both are set to 0.

These properties are deprecated now and use following properties for this purpose

mapreduce.map.failures.maxpercent
mapreduce.reduce.failures.maxpercent