How to merge Python Dask Dataframes into one on columns?

Question

got a little problem. I have two dask dataframes with following format:

#DF1.csv
DATE|EVENTNAME|VALUE

#DF2.csv
DATE|EVENTNAME0|EVENTNAME1|...|EVENTNAMEX

I want to merge the value from DF1.csv into DF2.csv, at time t (Date) and column (EventName). I use Dask at the moment, because i'm working with huge datesets ~50gb. I noticed that you can't use direct assignment of values in Dask. So i tried, dd.Series.where:

df[nodeid].where(time,value) => Result in an error (for row in df.iterrows():
#df2.loc[row[0],row[1][0]] =row[1][1])

i also tried a merge, but the resulting Dask dataframe had no partitions, which result in a MemoryError, because all datasets will be loaded into memory, if i use the .to_csv('data-*.csv') method. It should be easy to merge the dataframes, but i have no clue at the moment. Is there a Dask pro, that could help me out?

Edit:// This works well in pandas but not with dask:

for row in df.iterrows():
    df2.loc[row[0],row[1][0]] =row[1][1]

Tried something like that:

for row in df.iterrows():
    df2[row[1][0]] = df2[row[1][0]].where(row[0], row[1][1]) 
#Result in Error => raise ValueError('Array conditional must be same shape as '

Any ideas?

lkaupp lkaupp · Accepted Answer · 2019-04-17T08:40:38

For everyone who is interested, you can use:

#DF1
df.pivot(index='date', columns='event', values='value') #to create DF2 Memory efficient

see also: https://pandas.pydata.org/pandas-docs/stable/user_guide/reshaping.html

before, it took a huge time, was horrible memory hungry and brought up not the results that i was looking for. Just use Pandas pivot, if you try to alter your dataframe scheme.

Edit:// And there is no reason to use Dask anymore, speed up the whole process even further ;)

How to merge Python Dask Dataframes into one on columns?

1 Answers