Local spark and parquet files

Question

I have transaction information for individual transactions (eg. customer code, product, product group, price, etc.)

I have this now partitioned in parquet files for each year_month

This is very effective when reporting aggregates, etc. over product groups, etc.

However if I want to retrieve information for a specific customer across months this is not very effective/fast.

I tried to partition by year_month & customer_code, but then there's a lot of disk i/o since each partition is now a customer code with one line of data in it.

Is there a way to increase performance and let's say stick 10000 customers in one partition? Or say partition to the next group if the parquet file size is 64Mb or something like that.

With the logic in Spark that it has the min max per attribute in the parquet file I expect performance to boost, but I am too new to spark/parquet to really understand if this is a correct thoughtpiece and if it is technically possible. (of course I can create customer code groups myself and use this in querying too, but I was hoping something more automatically is possible).

Thanks,

G.

What about putting your data into cassandra and do the partitioning in there but with indexing. Would it make your random searches for customer per partition, faster? I understand that this is not an answer but I have experienced basically a similar situation before and used cassandra but I am not sure what would happen if I was using parquet! — M.Rez

kostya kostya · Accepted Answer · 2015-10-21T13:15:52

If you order data in each file by customer code and configure Spark to use parquet predicate pushdown (enabled by default) then full scan by customer code will be faster.

Internally parquet file stores column min/max values for each page and block. Filtering by value can efficiently skip reading pages and blocks based on this statistics.

Local spark and parquet files

1 Answers