Weka: Results of each fold in 10-fold CV

10
votes

For Weka Explorer (GUI), when we do a 10-fold CV for any given ARFF file, then what Weka Explorer provides (as far as I can see) is the average result for all the 10 folds.

Q. Is there any way to get the results of each fold? For instance, I need the error rates (incorrectly identified instances) for each fold.

Help appreciated.

2
machine-learningweka
why dont you use weka API instead? It will allow you much finer control. – iinception
@iinception, just wanted to see some insights of data (sanity test). So, trying to save some times of coding at this stage. – Rushdi Shams

2 Answers

16
votes

I think this is possible using Weka's GUI. You need to use the Experimenter though instead of the Explorer. Here are the steps:

  1. Open the Experimenter from the GUI Chooser
    • Create a new experiment (New button @ top-right)
    • [optional] Enter a filename and location in the Results Destination to save the results to
    • Set the Number of (cross-validation) folds to your liking (start experimenting with 2 folds for easy results)
    • Add your dataset (if your dataset needs preprocessing then you should do this in the Explorer first and then save the preprocessed dataset)
    • Set the Number of repetitions (I recommend 1 to start of with)
    • Add the algorithm(s) you want to test (again start easy, start with one algorithm)
  2. Go to the Run tab and Start the experiment and wait till it finishes
  3. Go to the Analyse tab and import the experiment results by clicking Experiment (top-right)
    • For Row select: Fold
    • For Column select: Percent_incorrect or Number_incorrect (or any other measure you want to see)
    • You now see the specified results for each fold
4
votes

Weka Explorer does not have an option to give the results for individual folds when using the crossvalidation option, there are some workarounds. If you explicitly don't want to change any code, you need to do some manual fiddling, but I think this gives more or less what you want

  1. Instead of Cross-validation, select Percentage split and set it to 90%
  2. Start classifier
  3. Click More options... and change the Random seed for XVal / % Split value to something you haven't used before.
  4. Repeat ten times.

This is not exactly equivalent to 10-fold crossvalidation though, since the pseudo-folds you make this way might overlap.

An alternative that is equivalent to crossvalidation, but more cumbersome, would be to make 10 folds manually by using the unsupervised instance filter RemoveFolds or RemoveRange. Generate and save 10 training sets and 10 test sets. Then for every fold, load the training set, select Supplied test set in the classify tab, and select the appropriate test fold.