Python Machine Learning - Training/Testing and apply prediction to new dataset

Question

I have only done training and testing on a single data set splits. I have a supervised learning problem: Data 1 Training/Testing and Data 2: No labels. I am using pandas data frame.

Data set 1: supervised

text        y_variable
apple       fruit
orange      fruit
celery      vegetable
mango       fruit

Data set 2: with no label

text        to_be_predicted
orange      ?
celery      ?
mango       ?

I am using scikit learn:

X = df['text']
y = df['y_variable']

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2

This splits the existing dataframe into training and testing. How to I train/test first Data set 1 and apply it to the second data set? Machine Learning.

Data set 2: with no label

text        to_be_predicted
orange      fruit
celery      vegetable
mango       fruit

Are you trying to train an supervised method and test as an unsupervised method?? + Don't you have extra information about each item? For example.... Shape / Weight / Color....?? Your df has only 2 columns? — spYder

Callin Callin · Accepted Answer · 2019-10-04T16:56:43

Many scikit-learn supervised classifiers have the ability to predict with new data.

For example, check out the documentation for K nearest neighbors:

knn.predict(new_data) # will predict classes for new data

UPDATE

When you predict the classes, based on new data, you only need to specify the new X. Here's a longer version of the code to describe better:

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt

# make some example data
X, y = make_blobs(n_samples = 100, n_features = 2, 
                  centers = 2, random_state = 123)

# fit supervised KNN classifier
knn = KNeighborsClassifier()
knn.fit(X, y) 

# create 50 new data points
# with the same number of features as the training set
new_data = np.random.randn(50, 2)

# predict new labels
new_labels = knn.predict(new_data)

# plot training clusters
plt.plot(X[y== 1, 0], 
         X[y==1,1], 
         "C1o", label = "training cluster 1")
plt.plot(X[y== 0, 0], 
         X[y==0,1], 
         "C0o", label = "training custer 2")

# plot predictions on new data
plt.plot(new_data[new_labels== 1, 0], 
         new_data[new_labels==1,1], 
         "ro", label = "new data assigned to cluster 1")
plt.plot(new_data[new_labels== 0, 0], 
         new_data[new_labels==0,1], 
         "bo", label = "new data assigned to cluster 2")
plt.legend()

Python Machine Learning - Training/Testing and apply prediction to new dataset

2 Answers