dask read multiple parquet files with different schemas

Question

I would like to read multiple parquet files with different schemes to pandas dataframe with dask, and be able to merge the schemes. When I talking about the different schemes, I mean, that there are common columns in all these files but in some files there are columns that are not present in others.

Unfortunately, when I read the files with

dd.read_parquet(my_parquet_files, engine="fastparquet")

I have only common columns read. I know that in spark there is a read option mergeSchema, I wonder if there is a simple way to do the same in dask ?

Answer 1

I recommend reading the different kinds of files individually, and then concatenating them with dd.concat

dfs = [dd.read_parquet(...) for ... in ...]
df = dd.concat(dfs, axis=0)

Then whatever policy Pandas uses for concatenating dataframes with mixed columns will take over. If Pandas supports this kind of behavior then Dask dataframe will likely support the behavior.

If it doesn't, then it sounds like you're asking for a feature request, in which case you should probably raise an issue at https://github.com/dask/dask/issues/new