why am I getting SettingWithCopyWarning from pandas

Question

I went through some questions regarding this warning, and am still a little bit confused. My method looks as follows:

    def _apply_standard_filters(self, df: pd.DataFrame) -> pd.DataFrame:
        logger.info("Applying standard filters")
        df["Type"] = df["Type"].apply(lambda x: x.strip().lower())

        df = df[df.Type == "xxxx"]

        df = df[df['Other Column'].str.contains(r"some_regex", na=False, regex=True)]

        return df

I'm getting warning regarding that line with lambda function. Initially I got the warning when line looked like this:

        df["Type"] = df["Type"].str.strip()
        df["Type"] = df["Type"].str.lower()

I thought maybe this is something with assigning twice the same column, so I rewrote to use apply (like in 1st snippet). Still getting the same warning. Finally I rewrote again with loc, like this:

df["Type"] = df.loc[:, "Type"].apply(lambda x: x.strip().lower())

and warning is still there. What am I missing here?

Also here is the output I'm getting

2020-08-12 15:38:14,498 - filters - INFO - Applying standard filters
filters.py:16: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df["Type"] = df.loc[:, "Type"].apply(lambda x: x.strip().lower())

Perhaps it is regarding something I do bfore that method is called? I'm using this _apply_standard_filters() always, after I use other filtering method depending on user's choice. However, I never touch Type or Other Column before standard filters. usually it looks like this:

df = df[df.Column1.str.contains(r"some other regex pattern", na=False, regex=True)]
df = self._apply_standard_filters(df=df)

Any suggestions are appreciated.

Cheers!

EDIT

After @r.ook answer I applied his solution, just fixed it a little bit and still have the same warning in front of my face. Now code looks like below:

def _apply_standard_filters(self, df: pd.DataFrame) -> pd.DataFrame:
    logger.info("Applying standard filters")
    df["Type"] = df.loc[:, "Type"].apply(lambda x: x.strip().lower())

    df = df.loc[df.Type == "xxxx"]

    df = df.loc[df['Other Column'].str.contains(r"some_regex", na=False, regex=True)]

    df = df[df['Other Column'].str.contains(r"some_regex", na=False, regex=True)]

    return df

def filter_something(self, df: pd.DataFrame) -> pd.DataFrame:
    df = df.loc[df.Column1.str.contains(r"^Some regex", na=False, regex=True)]
    df = self._apply_standard_filters(df=df)
    return df

When I run this I'm still getting:

filters.py:17: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df["Type"] = df.loc[:, "Type"].apply(lambda x: x.strip().lower())

Should I focus on this particular line mentioned in the warning? I haven't seen this warning before. It appeared when I added strip() and lower() to the df["Type"]. When I run the code w/o that line there is no warning, regardless of changes I made in other df assignments.

r.ook r.ook · Accepted Answer · 2020-08-12T14:16:28

Whenever you have lines that look like this df = df[...] it will trigger the SettingWithCopyWarning.

From the last of your question, it seems you are already passing a copy into the method, that's why the warning is triggered.

Do this instead:

df = df.loc[:, df.Column1.str.contains(r"some other regex pattern", na=False, regex=True)]
df = self._apply_standard_filters(df=df)

Likewise, edit all the lines in your method that follows the same df = df[...] pattern:

    df = df.loc[:, df.Type == "xxxx"]
    df = df.loc[:, df['Other Column'].str.contains(r"some_regex", na=False, regex=True)]

why am I getting SettingWithCopyWarning from pandas

1 Answers