Classfication accuracy on Weka

0
votes

I am using Weka GUI for a classification. I am new to Weka and getting confused with the options

  • Use training Set
  • Supplied test set
  • Cross validation

to train my classification algorithm (for example J48), I trained with cross validation 10 folds and the accuracy is pretty good (97%). When I test my classification - the accuracy drops to about 72%. I am so confused. Any tips please? This is how I did it:

  • I train my model on the training data (For example: train.arff)
  • I right-click in the Results list on the item which model you want to save select Save model and save it for example as j48tree.model

and then

  • I load the test data (for example: test.arff via the Supplied test set button
  • Right-click in the Results list, I selected Load model and choose j48tree.model
  • I selected Re-evaluate model on current test set

Is the way i do it wrong? Why the accuracy miserably dropping to 72% from 97%? Or is doing only the cross-validation with 10 folds is enough to train and test the classifier?

Note: my training and testing datasets have the same attributes and labels. The only difference is, I have more data on the testing set which I don't think will be a problem.

1
machine-learningclassificationweka

1 Answers

1
votes

I don't think there is any issue with how you use WEKA.

You mentioned that you test set is larger than training? What is the split? The usual rule of thumb is that test set should be one 1/4 of the whole dataset, i.e. 3 times smaller than training and definitely not larger. This alone could explain the drop from 97% to 72% which is by the way not so bad for real life case.

Also it will be helpful if you build the learning curve https://weka.wikispaces.com/Learning+curves as it will explain whether you have a bias or variance issue. Judging by your values sounds like you have a high variance (i.e. too many parameters for your dataset), so adding more examples or changing your split between training and test set will likely help.

Update I ran a quick analysis of the dataset at question by randomforest and my performance was similar to the one posted by author. Details and code are available on gitpage http://omdv.github.io/2016/03/10/WEKA-stackoverflow