python - Pandas groupby multiple fields then diff

Question

So my dataframe looks like this:

from pandas.compat import StringIO
d = StringIO('''
date,site,country,score
2018-01-01,google,us,100
2018-01-01,google,ch,50
2018-01-02,google,us,70
2018-01-03,google,us,60
2018-01-02,google,ch,10
2018-01-01,fb,us,50
2018-01-02,fb,us,55
2018-01-03,fb,us,100
2018-01-01,fb,es,100
2018-01-02,fb,gb,100
''')

df = pd.read_csv(d, sep=",")

Each site has a different score depending on the country. I'm trying to find the 1/3/5-day difference of scores for each site/country combination.

Output should be:

date,site,country,score,1_day_diff
2018-01-01,google,ch,50,0
2018-01-02,google,ch,10,-40
2018-01-01,google,us,100,0
2018-01-02,google,us,70,-30
2018-01-03,google,us,60,-10
2018-01-01,fb,es,100,0
2018-01-02,fb,gb,100,0
2018-01-01,fb,us,50,0
2018-01-02,fb,us,55,5
2018-01-03,fb,us,100,45

I first tried sorting by site/country/date, then grouping by site and country but I'm not able to wrap my head around getting a difference from a grouped object.

how to i get StringIO for python3? I am trying to recreate your problem — Julian Rachman
@Alex @ayhan I've edited the expected output. Essentially, es and gb would appear before us. — Craig
@Craig for that you could add df.sort_values(by=['site', 'country', 'date'], ascending=[False, True, True]) with @ayhan 's answer — akhilsp

ayhan ayhan · Accepted Answer · 2018-01-19T19:11:20

First, sort the DataFrame and then all you need is groupby.diff():

df = df.sort_values(by=['site', 'country', 'date'])

df['diff'] = df.groupby(['site', 'country'])['score'].diff().fillna(0)

df
Out: 
         date    site country  score  diff
8  2018-01-01      fb      es    100   0.0
9  2018-01-02      fb      gb    100   0.0
5  2018-01-01      fb      us     50   0.0
6  2018-01-02      fb      us     55   5.0
7  2018-01-03      fb      us    100  45.0
1  2018-01-01  google      ch     50   0.0
4  2018-01-02  google      ch     10 -40.0
0  2018-01-01  google      us    100   0.0
2  2018-01-02  google      us     70 -30.0
3  2018-01-03  google      us     60 -10.0

sort_values doesn't support arbitrary orderings. If you need to sort arbitrarily (google before fb for example) you need to store them in a collection and set your column as categorical. Then sort_values will respect the ordering you provided there.

python - Pandas groupby multiple fields then diff

1 Answers