convert text columns into numbers in sklearn

Question

I'm new to data analytics. I'm trying some models in python Sklearn. I have a dataset in which some of the columns have text columns. Like below,

Dataset

Is there a way to convert these column values into numbers in pandas or Sklearn?. Assigning numbers to these values will be right?. And what if a new string pops out in test data?.

Please advice.

consider using get_dummies function available in pandas. Ignore all new values encountered in test data, you cannot use values which was not seen in during training. — shanmuga
i was thinking of using it. but some of the columns have many unique values (upto 400+). — Selva Saravana Er

Amir F Amir F · Accepted Answer · 2017-03-11T09:08:16

Consider using Label Encoding - it transforms the categorical data by assigning each category an integer between 0 and the num_of_categories-1:

from sklearn.preprocessing import LabelEncoder
df = pd.DataFrame(['a','b','c','d','a','c','a','d'], columns=['letter'])

  letter
0      a
1      b
2      c
3      d
4      a
5      c
6      a

Applying:

le = LabelEncoder()
encoded_series = df[df.columns[:]].apply(le.fit_transform)

encoded_series:

convert text columns into numbers in sklearn

2 Answers