Understanting file distribution and partitioning in HDFS when using Hive

Question

On the one hand, in HDFS docs they say:

HDFS is designed to support very large files. Applications that are compatible with HDFS are those that deal with large data sets. These applications write their data only once but they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS supports write-once-read-many semantics on files. A typical block size used by HDFS is 64 MB. Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a different DataNode.

Meaning every file will be splitted between nodes.

On the other hand, when I use Hive or Spark SQL, I manage the partitions in such a way that there is a folder for each partition, and all the files inside belong to this partition. For example:

/Sales
   /country=Spain
      /city=Barcelona
         /2019-08-28.parquet
         /2019-08-27.parquet
      /city=Madrid
         /2019-08-28.parquet
         /2019-08-27.parquet

Let's say that each file's size is 1GB and the HDFS block size is 128 MB.

So I am confused. I don't understand if city=Barcelonav/2019-08-28.parquet is saved on only one node as a whole (even together with city=Barcelona/2019-08-27.parquet), or each file is distributed between 8 nodes.

If each file is distributed, then what is the benefit of the partitions?

If the data is distributed according to the partitions I define, how does HDFS know to do that? Does it look for folders with a name in the form of key=value and make sure they will be saved intact?

Vihit Shah Vihit Shah · Accepted Answer · 2019-08-28T16:13:03

You are confused between "how HDFS stores the files that we dump into it" and "how Hive/Spark creates different directories in case of partitioning".

Let me try to provide you a perspective. HDFS works as you have mentioned. HDFS breaks up the files into n number of blocks depending upon the block size and the size of the file to be stored. The metadata (directories, permissions, etc..) is an abstraction in a sense that the file (2019-08-27.parquet) that you see as one is indeed distributed among nodes. Namenode maintains the metadata.

However, when we partition it creates different directories on HDFS. This ultimately helps when you want to query the data with conditions on the partitioned column. Only relevant directories are searched for the requested data. If you go ahead and query on your partitioned data and write an explain to have a look at the logical plan, you can notice the Partition Filters while FileScan phase.

The partitioned data is still stored on HDFS in the same way that you mentioned.

Hope this helps!

Understanting file distribution and partitioning in HDFS when using Hive

1 Answers