Does dask load all columns when doing an operation on a single column?

Question

Whenever I do computations using a dask dataframe I make sure I'm only loading the necessary columns to be able to save on computation speed.

I just don't understand how dask works internally and why doesn't he benefit from parquet columnar format.

In the small example below test.parquet is a parquet file with 13 columns of various dtypes, 10M rows and 16 partitions. As you can see, if I'm only interested in the min of a single column I have a very noticeable speed increase when I load just the target column. When using the distributed scheduler I can also see that less data is loaded into memory.

Why does dask take longer to compute when I load the entire dataframe vs only target columns? What explains the overhead in the same computation as seen in the example?

Answer 1

This is actually a performance regression which we are working to correct. Code has existed to pass on column selections from the dataframe down to the loader automatically to do what you are doing explicitly, but the newer graph optimisation code does not yet work fully with the parquet IO pieces. If your workaround is too annoying, you may get the same performance by downgrading Dask a couple of point releases.