Pandas: Transpose, groupby and summarize columns

1
votes

i have a pandas DataFrame which looks like this:

| Id | Filter 1 | Filter 2 | Filter 3 |
|----|----------|----------|----------|
| 25 | 0        | 1        | 1        |
| 25 | 1        | 0        | 1        |
| 25 | 0        | 0        | 1        |
| 30 | 1        | 0        | 1        |
| 31 | 1        | 0        | 1        |
| 31 | 0        | 1        | 0        |
| 31 | 0        | 0        | 1        |

I need to transpose this table, add "Name" column with the name of the filter and summarize Filters column values. The result table should be like this:

| Id | Name     | Summ |
| 25 | Filter 1 | 1    |
| 25 | Filter 2 | 1    |
| 25 | Filter 3 | 3    |
| 30 | Filter 1 | 1    |
| 30 | Filter 2 | 0    |
| 30 | Filter 3 | 1    |
| 31 | Filter 1 | 1    |
| 31 | Filter 2 | 1    |
| 31 | Filter 3 | 2    |

The only solution i have came so far was to use apply function on groupped by Id column, but this mehod is too slow for my case - dataset can be more than 40 columns and 50_000 rows, how can i do this with pandas native methods?(eg Pivot, Transpose, Groupby)

3
pythonpandasdataframe

3 Answers

2
votes

Use:

df_new=df.melt('Id',var_name='Name',value_name='Sum').groupby(['Id','Name']).Sum.sum()\
                                                                 .reset_index()
print(df_new)

   Id      Name  Sum
0  25  Filter 1    1
1  25  Filter 2    1
2  25  Filter 3    3
3  30  Filter 1    1
4  30  Filter 2    0
5  30  Filter 3    1
6  31  Filter 1    1
7  31  Filter 2    1
8  31  Filter 3    1
1
votes

stack then groupby 

df.set_index('Id').stack().groupby(level=[0,1]).sum().reset_index()
   Id   level_1  0
0  25  Filter 1  1
1  25  Filter 2  1
2  25  Filter 3  3
3  30  Filter 1  1
4  30  Filter 2  0
5  30  Filter 3  1
6  31  Filter 1  1
7  31  Filter 2  1
8  31  Filter 3  1

Short version 

df.set_index('Id').sum(level=0).stack()#df.groupby('Id').sum().stack()
0
votes

Using filter and melt

df.filter(like='Filter').groupby(df.Id).sum().T.reset_index().melt(id_vars='index')

    index       Id  value
0   Filter 1    25  1
1   Filter 2    25  1
2   Filter 3    25  3
3   Filter 1    30  1
4   Filter 2    30  0
5   Filter 3    30  1
6   Filter 1    31  1
7   Filter 2    31  1
8   Filter 3    31  2