Cross Validation in Classification

1
votes

I have two different datasets, datset X and dataset Y... From which I calculate features to use for classification..

Case 1. When I combine both together as one large datset then use 10 fold cross validation I get very good classification results with accuracy and AUC > 95%

Case2. Yet if I use one of the datasets for training and the other for testing, results fall severely low with both accuracy and AUC becoming ~ 50%

My questions are:

  1. Which of the cases' results is more reliable??

  2. And why the huge difference in results??

Thanks..

1
image-processingclassificationcross-validation
did I answer your question? Do you need any extra help? - João Almeida

1 Answers

1
votes

There could be a bias in the way the datasets were obtained that makes you get worst results.

Read this.

Another thing is that on one case you are training your classifier with a smaller dataset (the two combined is larger assuming they are about the same size, even with the 10 fold cross validation). This necessarily causes a poorer performance.

So my answers would be:

  1. Depends on how you obtained both datasets and on how the final classifier will be used.

  2. Differences in the size of the training set and bias on how they are obtained.