Does Matlab's crossval method respect class frequencies?

Question

Does Matlab's crossval method in the context of (binary) classification respect class frequencies?

Most classification models in Matlab offer the possibility to compute a cross-validated model. For instance when training a linear SVM by means of svm=fitcsvm(X,y);, one can compute a cross-validated model by calling cv=crossval(svm);. (Here the documentation for the method crossval for objects of type ClassificationSVM.) This cross-validated model then can be used to estimate the generalization error of the training process.

Now my question(s): When partitioning the training data, does crossval take into account the class frequencies? For instance, we may have 5 times more observations $X_0$ for class 0 than observations $X_1$ for class 1. So do the partitioned versions of the data roughly have the same ratio of observations for each class (5:1 in my example)? Or is this completely ignored, with the reason that if the dataset is large enough, the partitions very likely will have about the same relative class size.

Before dealing with Matlab's crossval feature, I used my own partitioning algorithm, that respected the relative class sizes when splitting the data. In essence, the algorithm would draw 5 items of class 0 at random and then 1 item of class 1 if the class frequencies were 5/6 and 1/6, until the partitions were full.

If the the relative class sizes were ignored, I'd say that this can be problematic for very imbalanced and/or small datasets. Or am I mistaken here? Very glad to read your thoughts about this.

deemel deemel · Accepted Answer · 2018-01-30T10:34:33

If the the relative class sizes were ignored, I'd say that this can be problematic for very imbalanced and/or small datasets.

Yes it can become problematic, one of the cases that you can run into is simply not encountering instances of a particular minority class at all in one fold of the crossvalidation. Since you're dealing with binary classification I would assume this becomes less of an issue, but depending on how strong the class imbalance is one might still get unlucky in one fold.
However please note that if you're dealing with imbalanced data, you might give the matter a read and for example consider the class imbalance in your evaluation measure.

Regarding your first question

What you described as 'respecting the class frequencies' is usually achieved via a stratified sample. In the case that you described Matlabs crossval function does not automatically use stratified partitions as part of the crossvalidation. However, they provide an example for this in the documentation for crossval , example 2:

load('fisheriris');
y = species;
X = meas;
cp = cvpartition(y,'k',10); % Stratified cross-validation

classf = @(XTRAIN, ytrain,XTEST)(classify(XTEST,XTRAIN,...
ytrain));



cvMCR = crossval('mcr',X,y,'predfun',classf,'partition',cp)
cvMCR =
    0.0200

What you see here is them creating a 'partitioning scheme' for the data based on the variable y via the cvpartition class. In this case the partitioning will create 10 partitions and try to keep the ratios of the different y values roughly equal throughout all these partitions.

You should have a look at the documentation of cvpartition, too, since there they describe the current syntax , which apparently changed since the upper example was created. Based on the current one it should be the following for k=10

c = cvpartition(y,'KFold',10)

I hope that answered your questions. Please let me know if I missed anything.

Does Matlab's crossval method respect class frequencies?

1 Answers