python - pandas fillna not working

22
votes

I have a dataframe with nans in it:

>>>df.head()
Out[1]: 
            JPM US SMALLER COMPANIES C ACC
1990-01-02                             NaN
1990-01-03                             NaN
1990-01-04                             NaN
1990-01-05                             NaN
1990-01-08                             NaN

I have another dataframe with values in it:

>>>t.head()
Out[1]: 
1990-01-02    51.95
1990-01-03    52.63
1990-01-04    53.04
1990-01-05    52.07
1990-01-08    51.73
Name: JPM US SMALLER COMPANIES C ACC, dtype: float64

Unfortunately, df.fillna does not appear to be working for me:

>>>df.fillna( t ).head()
Out[1]: 
            JPM US SMALLER COMPANIES C ACC
1990-01-02                             NaN
1990-01-03                             NaN
1990-01-04                             NaN
1990-01-05                             NaN
1990-01-08                             NaN

[5 rows x 1 columns]

Why is this happening? I am on pandas 0.13.1

4
pythonpandas
Oops. I fixed it. df should have been a series, not a dataframe. After that it worked. - Ginger
In the case where you are using a DataFrame, you can use DataFrame.where to use another frame's values to replace the values when null. - benjwadams
@benjwadams I think in that situation it might be better to use combine_first or update. They're custom built for that purpose, so they should help you avoid errors. - Roger Fan

4 Answers

62
votes

you need inplace

df[1].fillna(0, inplace=True)
5
votes

You need to assign the value df = df.fillna( t )

2
votes

You have two options:

1) Specific for each column

cols_fillna = ['column1','column2','column3']
# replace 'NaN' with zero in these columns
 for col in cols_fillna:
     df[col].fillna(0,inplace=True)
     df[col].fillna(0,inplace=True)

2) For the entire dataframe

df = df.fillna(0)
1
votes

Alternativly:

df = df.replace(np.nan, 0) #or any other value you deem fit

df.replace(np.nan, 0) or df.fillna(0) threw me off when I applied certain str.replace()-operations right after Na-operations.. so watch out for your order of commands -> first str.replace() than fillna()