Hadoop sort phase taking hours

0
votes

I started using hadoop for a week. After succesfully running the examples, I a mapreduce job to find the most used word using the WordCount example.

I'm trying to run this job with 500 MB or data.

However the map task is taking hours. it's currently at map 67% reduce 0%.

The map task log is here as below:

2014-10-24 11:19:52,274 DEBUG [IPC Parameter Sending Thread #0] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:52026 from job_1414134493988_0001 sending #2554
2014-10-24 11:19:52,278 DEBUG [IPC Client (592959754) connection to /xxx.xx.xx.xx:52026 from job_1414134493988_0001] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:52026 from job_1414134493988_0001 got value #2554
2014-10-24 11:19:52,279 DEBUG [communication thread] org.apache.hadoop.ipc.RPC: Call: ping 5
2014-10-24 11:19:55,279 DEBUG [IPC Parameter Sending Thread #0] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:52026 from job_1414134493988_0001 sending #2555
2014-10-24 11:19:55,280 DEBUG [IPC Client (592959754) connection to /xxx.xx.xx.xx:52026 from job_1414134493988_0001] org.apache.hadoop.ipc.Client: IPC Client (592959754) connection to /xxx.xx.xx.xx:52026 from job_1414134493988_0001 got value #2555
2014-10-24 11:19:55,280 DEBUG [communication thread] org.apache.hadoop.ipc.RPC: Call: ping 1

Is this suppose to happen, take that long?

1
javasortinghadoopmapreduce

1 Answers

0
votes

Some tips:

  • what do you mean by long (how long)?
  • Take a stack trace where the map task is running for long time and see where it's stucked?
  • Also what's the state of tasks are they failing frequently?
  • How many map and reduce in your cluster?