Create training and test dataset in Pandas

Question

I would like to manually create a training and test dataset in pandas rather than use cross-validation from sklearn. I am nearly successful. However, I find discrepancy in the numbers between df_training and df_test. Why is that?

Here's what I did:

Created a new dataset called df_training from original data frame by selecting rows at random with random.choice
Created a new dataset called df_test by dropping rows from the original dataset that are in df_training using the df.drop(df_training.index) as the argument.

This is where I do not get the correction dimensions of df_test when the dimensions of df and df_training remain constant.

from sklearn.datasets import load_boston 
boston = load_boston()

names = ['crim', 'zn', 'indus', 'chas', 'nox', 'rm', 'age', 'dis', 'rad', 'tax', 'ptratio', 'b', 'lstat']
df = pd.DataFrame(boston.data, columns=names)
# add in prices
df['price'] = boston.target

df.shape
(506, 14)

import random
# Use 70% of the DataFrame and call is df_training
df_training = df.ix[np.random.choice(df.index, 354)]
df_training.shape

# Remove the 70% of data from the main DataFrame and call it df_test
df_test = df.drop(df_training.index)

df_test.shape
(250, 14)

Should I not get 504 - 354 = 150?

Interestingly when I run whole code couple of times I get different results for the test_set. Shouldn't I get the same result when training set and original set are constant? What's going on?

In [26]: %run create_training.py
Original Set:  (506, 14)
training set:  (354, 14)
test set:  (247, 14)

In [27]: %run create_training.py
Original Set:  (506, 14)
training set:  (354, 14)
test set:  (254, 14)

In [28]: %run create_training.py
Original Set:  (506, 14)
training set:  (354, 14)
test set:  (241, 14)

Thanks I will check that out. I just wanted to use something that is not part of sklearn. — Rohit

Matt Matt · Accepted Answer · 2015-05-06T07:07:01

I think the two missing ingredients here are:

Setting the seed for numpy random functions in order to make the split reproducible.
Calling np.random.choice using replacement=False (refer to the docs for more information).

The code:

# make results reproducible
np.random.seed(42)
# sample without replacement
train_ix = np.random.choice(df.index, 354, replace=False)
df_training = df.ix[train_ix]
df_test = df.drop(train_ix)

Create training and test dataset in Pandas

1 Answers