Why is Drill fastest with one partition?

Question

My cluster has 6 nodes, each with 2 cores. I have a Spark job saving a Parquet file of the size of ~150MB to HDFS. If I repartition my dataframe to 6 partitions before saving, Drill queries are actually 30-40% slower than when I repartition it to 1 partition. Why is that? Is it expected? Can it indicate an issue with my setup?

Update

Results of the same SQL query in seconds (3 runs per number of partitions)

1 partition: 1.238, 1.29, 1.404
2 partitions: 1.286 1.175 1.259
3 partitions: 1.699 1.8 1.7
6 partitions: 2.223, 1.96, 1.772
12 partitions: 1.311, 1.335, 1.339
24 partitions: 1.261 1.302 1.235
48 partitions: 1.664 1.757 2.133

As you can see 1, 2, 12 and 24 partitions are fast. 3, 6 and 48 partitions are very clearly slower. What could be causing that?

Answer 1

When saving the parquet file in spark using a single partition, you would save the file locally to the partition on a single node. Once that happens, replication needs to kick in and distribute the file over the different nodes.

When saving the partquet file in spark using multiple partitions, spark would save the file distributed already, however, maybe not exactly the way HDFS needs it. Still replication and re-distribution needs to kick-in, but now in a much complexer situation.

Then depending on your spark process, you might have your data already sorted in a different way (1 vs multiple partitions), potentially making it more suitable for the next process in line (Drill).

I really cannot pintpoint a reason, but with such a small difference in time (you are talking seconds), I am not sure if the difference is even distinctive enough.

Then also, we might need to put the test methods in doubt. Java Garbage collection, Background Processes running (including the replicaton process), etc etc.

One suggestion I would have is to leave your HDFS cluster at rest for a while to make sure replication and other processes quiet down before you start with the Drill process.