HDFS performance for small files

Question

I'm new to Hadoop. Recently I'm trying to process (only read) many small files on hdfs/hadoop. The average file size is about 1 kb and the number of files is more than 10M. The program must be written in C++ due to some limitations.

This is just a performance evaluation so I only use 5 machines for data nodes. Each of the data node have 5 data disks.

I wrote a small C++ project to read the files directly from hard disk(not from HDFS) to build the performance base line. The program will create 4 reading threads for each disk. The performance result is to have about 14MB/s per disk. Total throughput is about 14MB/s * 5 * 5 = 350MB/s (14MB/s * 5 disks * 5 machines ).

However, when this program ( still using C++, dynamically linked to libhdfs.so, creating 4*5*5=100 threads) reads files from hdfs cluster, the throughput is about only 55MB/s.

If this programming is triggered in mapreduce (hadoop streamming, 5 jobs, each have 20 threads, total number of threads is still 100), the throughput goes down to about 45MB/s. (I guess it's slow down by some bookkeeping process).

I'm wondering what is the reasonable performance HDFS can prvoide. As you can see, comparing with native code, the data throughput is only about 1/7. Is it the problem of my config? Or HDFS limitation? Or Java limitation? What's the best way for my scenario? Will sequence file help (much)? What is the reasonable throughput comparing to native IO read we can expect?

Here's some of my config:

NameNode heap size 32G.

Job/Task node heap size 8G.

NameNode Handler Count: 128

DataNode Handler Count: 8

DataNode Maximum Number of Transfer Threads: 4096

1GBps ethernet.

Thanks.

Supplemental: The program read a file list from stdin, which contains millions of file paths. — avhacker
I always forget the "why's" and "how's," but try making your input files at least as large as the block size (default 64 MB) and then re-run your profiling. The way you combine your files depends on their format; like you can just concatenate them if they're simply text. — Matt D
I know that merge the files into bigger files can significantly improve the performance but this won't be our first choice. BTW, reading files directly from disk will have much improvement, too. I really like to know what's the reasonable throughput HDFS can provide comparing to native access. 1/7 doesn't seem good. — avhacker

Joe K Joe K · Accepted Answer · 2012-12-21T18:29:48

HDFS is really not designed for many small files.

For each new file you read, the client has to talk to the namenode, which gives it the location(s) of the block(s) of the file, and then the client streams the data from the datanode.

Now, in the best case, the client does this once, and then finds that it is the machine with the data on it, and can read it directly from disk. This will be fast: comparable to direct disk reads.

If it's not the machine that has the data on it, then it must stream the data over the network. Then you are bound by network I/O speeds, which shouldn't be terrible, but still a bit slower than direct disk read.

However, you're getting an even worse case- where the overhead of talking to the namenode becomes significant. With only 1KB files, you are getting to the point where you're exchanging just as much metadata as actual data. The client has to make two separate network exchanges to get the data from each file. Add to this that the namenode is probably getting hammered by all of these different threads and so it might become a bottleneck.

So to answer your question, yes, if you use HDFS for something it's not designed to be used for, it's going to be slow. Merge your small files, and use MapReduce to get data locality, and you'll have much better performance. In fact, because you'll be able to take better advantage of sequential disk reads, I wouldn't be surprised if reading from one big HDFS file was even faster than reading many small local files.

HDFS performance for small files

3 Answers