I started playing with hadoop 2.6.0, and set up a pseudo-distributed single-node system according to the official documentation.
When I run the simple Map Reduce (MR1) example (see "Pseudo-Distributed Operation -> Execution"), then the overall execution time is approx. 7 sec. More precise, bash's time gives:
real 0m6.769s user 0m7.375s sys 0m0.400s
When I run the same example via Yarn (MR2) (see "Pseudo-Distributed Operation -> YARN on Single Node"), then the overall execution time is approx. 100 sec , hence extremely slower. bash's time gives:
real 1m38.422s user 0m4.798s sys 0m0.319s
Hence, there is (for some reason) a large overhead outside userspace. But why?
Both examples were executed via
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'
Here more details for pure Map Reduce (MR1):
(...)
15/04/10 21:12:17 INFO mapreduce.Job: Counters: 38
File System Counters
FILE: Number of bytes read=125642
FILE: Number of bytes written=1009217
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=154548
HDFS: Number of bytes written=1071
HDFS: Number of read operations=157
HDFS: Number of large read operations=0
HDFS: Number of write operations=16
Map-Reduce Framework
Map input records=11
Map output records=11
Map output bytes=263
Map output materialized bytes=291
Input split bytes=129
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=291
Reduce input records=11
Reduce output records=11
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1062207488
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=437
File Output Format Counters
Bytes Written=197
real 0m6.769s
user 0m7.375s
sys 0m0.400s
Here more details for Yarn (MR2):
(...)
15/04/10 21:20:31 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=291
FILE: Number of bytes written=211001
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=566
HDFS: Number of bytes written=197
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=2411
Total time spent by all reduces in occupied slots (ms)=2717
Total time spent by all map tasks (ms)=2411
Total time spent by all reduce tasks (ms)=2717
Total vcore-seconds taken by all map tasks=2411
Total vcore-seconds taken by all reduce tasks=2717
Total megabyte-seconds taken by all map tasks=2468864
Total megabyte-seconds taken by all reduce tasks=2782208
Map-Reduce Framework
Map input records=11
Map output records=11
Map output bytes=263
Map output materialized bytes=291
Input split bytes=129
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=291
Reduce input records=11
Reduce output records=11
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=68
CPU time spent (ms)=1160
Physical memory (bytes) snapshot=432250880
Virtual memory (bytes) snapshot=1719066624
Total committed heap usage (bytes)=353370112
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=437
File Output Format Counters
Bytes Written=197
real 1m38.422s
user 0m4.798s
sys 0m0.319s
Can anybody explain this performance gap and how to fix it?
