3
votes

I have a pandas dataframe indexed by date. Let's assume it from Jan-1 to Jan-30. I want to split this dataset into X_train, X_test, y_train, y_test but I don't want to mix the dates so I want the train and test samples to be divided by a certain date (or index). I'm trying

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

But when I check the values, I see the dates are mixed. I want to split my data as:

Jan-1 to Jan-24 to train and Jan-25 to Jan-30 to test (as test_size is 0.2, that makes 24 to train and 6 to test)

How can I do this? Thanks

2
you should read documentNihal
if you want top 24 then use x.head(24) and for last 6 use x.tail(6) no need for train_test splitNihal
@Nihal random_state=None doesn't work. Tried that..iso_9001_
random_state=None will take numpy.random that's why it won't workNihal
Are you looking for TimeSeriesSplit? scikit-learn.org/stable/modules/generated/…FlorianGD

2 Answers

2
votes

you should use

X_train, X_test, y_train, y_test = train_test_split(X,Y, shuffle=False, test_size=0.2, stratify=None)

don't use random_state=None it will take numpy.random

in here its mentioned that use shuffle=False along with stratify=None

1
votes

Try using TimeSeriesSplit:

X = pd.DataFrame({'input_1': ['a', 'b', 'c', 'd', 'e', 'f'],
                  'input_2': [1, 2, 3, 4, 5, 6]},
                 index=[pd.datetime(2018, 1, 1),
                        pd.datetime(2018, 1, 2),
                        pd.datetime(2018, 1, 3),
                        pd.datetime(2018, 1, 4),
                        pd.datetime(2018, 1, 5),
                        pd.datetime(2018, 1, 6)])
y = np.array([1, 0, 1, 0, 1, 0])

Which results in X being

           input_1  input_2
2018-01-01       a        1
2018-01-02       b        2
2018-01-03       c        3
2018-01-04       d        4
2018-01-05       e        5
2018-01-06       f        6
tscv = TimeSeriesSplit(n_splits=3)
for train_ix, test_ix in tscv.split(X):
    print(train_ix, test_ix)
[0 1 2] [3]
[0 1 2 3] [4]
[0 1 2 3 4] [5]