pandas warning when using loc

Question

I have this code sample which supposed to split a dataframe into smaller dataframes using group-by and modify the smaller dataframes if the numbers in 'num' column are even:

import pandas as pd

df = pd.DataFrame({
    'id1': [1]*5+[2]*5,
    'num': range(11, 21),
    'x': range(10)

})
print df
for id1, grouped_df in df.groupby('id1'):
    grouped_df.loc[grouped_df['num'] % 2 == 0, 'num'] = 'even'
    print grouped_df

print df

however when I run this code I encounter the warning:

/usr/local/lib/python2.7/dist-packages/pandas/core/indexing.py:537: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

and the code is extremely slow when ran on "real" data.

According to what I understand, this warning suggests the use of loc but I am already using it! Does pandas view the dataframes returned by group-by as sliced? How can I get rid of this warning and ensure expected behavior?

It's not necessarily the case that the warning is connected to your program performance. A warning is a warning, it's often a false positive. You can try benchmarking to see what's taking time. — jpp

jezrael jezrael · Accepted Answer · 2018-06-14T11:56:51

Use copy:

for id1, grouped_df in df.groupby('id1'):
    grouped_df = grouped_df.copy()
    grouped_df.loc[grouped_df['num'] % 2 == 0, 'x'] = 'even'
    print (grouped_df)

   id1  num     x
0    1   11     0
1    1   12  even
2    1   13     2
3    1   14  even
4    1   15     4
   id1  num     x
5    2   16  even
6    2   17     6
7    2   18  even
8    2   19     8
9    2   20  even

pandas warning when using loc

1 Answers