How can i solve the value too large for dtype('float32') error? Pandas

Question

I have a dataset that contains numbers. But when I try to fit with this command:

model.fit(X_train, y_train)

I get this error :

Input contains NaN, infinity or a value too large for dtype('float32').

But there are no null cells in my dataset. Not scaled data_X

[[0 0 4 ... 0 -21.4 6]
 [1 0 2 ... 0 0.0 0]
 [0 0 2 ... 0 805.9 7]
 ...
 [1 0 2 ... 1 -20.2 0]
 [1 0 3 ... 1 1031.0 5]
 [0 1 3 ... 1 0.0 0]]

scaled X_train is like this:

 [[ 0.64649731 -0.63390308  0.74842646  0.41698984 -0.65263096]
 [-1.54679684 -0.63390308 -0.6061627   0.41698984 -0.65263096]
 [ 0.64649731  1.57752823  0.07113188  0.41698984  0.7140774 ]
 ...
 [ 0.64649731 -0.63390308 -0.6061627   0.41698984 -0.65263096]
 [-1.54679684 -0.63390308  0.07113188  0.41698984  0.7140774 ]
 [ 0.64649731 -0.63390308  0.74842646  0.41698984 -0.65263096]]

When I drop float numbers from this dataset, the problem solved. But this is not the solution. What can I do to avoid this?

Thanks.

What is your largest and smallest value? is it too large/small for float32??? — MattR
The error already suggest what to do: imputation, i.e. you need to replace NaN with some other values. — Quang Hoang
Sure i fixed self.data[self.data==np.inf]=np.nan self.data.fillna(0, inplace=True) — Şevval Kahraman

Renaud Renaud · Accepted Answer · 2020-01-22T15:25:00

Try to change value for float 32:

X_train= np.float32(X_train)

And/or replace Nan and inf:

X_train=np.nan_to_num(X_train, nan=-9999, posinf=33333333, neginf=33333333)

How can i solve the value too large for dtype('float32') error? Pandas

1 Answers