Parquet predicate pushdown

8
votes

Does parquet's predicate pushdown mean that only the data that is required is actually loaded from disk?

E.g. If I create a spark dataframe and only select particular fields, will only those fields be read from disk?

1
hadoopapache-sparkparquetbigdata
Kind of. Spark projects to get only required columns if it can but it is not what is predicate pushdown about. Predicate pushdown in generally covers conditions which are placed in the WHERE clause. - zero323

1 Answers

14
votes

Predicate pushdown deals with what values will be scanned and not what columns. So, if you apply filter on column A to only return records with value V, the predicate push down will make parquet read only blocks that may contain values V. Parquet holds min/max statistics in several levels, and it will compare the value V to the those min/max headers, and only scan blocks where min/max contains the value V. This is for predicate push down.

Another thing with parquet is "projection pushdown" - it stores data in columns, so when your projection limits the query to certain columns, only those columns will be returned. This feature is not what is called predicate pushdown though.