scikit learn: 5 fold cross validation & train test split

Question

I am wondering when I do train test split (20% test, 80% 80%) and then I apply 5 fold cross validation does that mean all data has been in the test set once? or is it randomly choosed each time that, in each fold the same events were possibly included in the test more than once and some possibly were never included in test set?

#20% of the data will be used as test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=seed) 

cv_results= cross_val_score(model, X_train, y_train, cv=kfold, scoring=scoring)

Grr Grr · Accepted Answer · 2018-06-28T15:06:16

Will all data have been in the test set once? Yes, at least within the data that you pass to your cross validation method. For example:

X = np.arange(10)
y = np.concatenate((np.ones(5), np.zeros(5)))
X
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
y
array([1., 1., 1., 1., 1., 0., 0., 0., 0., 0.])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2, random_state=42)
X_train
array([5, 0, 7, 2, 9, 4, 3, 6])
X_test
array([8, 1])
kf = Kfold(n_splits=5)
for train, test in kf.split(X_train):
    print(train, test)
[2 3 4 5 6 7] [0 1]
[0 1 4 5 6 7] [2 3]
[0 1 2 3 6 7] [4 5]
[0 1 2 3 4 5 7] [6]
[0 1 2 3 4 5 6] [7]

As you can see the indices for the test set run from 0 to 7 meaning that all 8 values in X_train will have appeared in the cross validation test once. This pattern would persist regardless of your sample size.

The size of the splits created by the cross validation split method are determined by the ratio of your data to the number of splits you choose. For example if I had set KFold(n_splits=8) (the same size as my X_train array) the test set for each split would comprise a single data point.

scikit learn: 5 fold cross validation & train test split

1 Answers