SVM poor performance compared to Random Forest

Question

I am using the scikit-learn library for python for a classification problem. I used RandomForestClassifier and a SVM (SVC class). However while the rf achieves about 66% precision and 68% recall the SVM only gets up to 45% each.

I did a GridSearch for the parameters C and gamma for the rbf-SVM and also considered scaling and normalization in advance. However I think the gap between rf and SVM is still too large.

What else should I consider to get an adequate SVM performance?

I thought it should be possible to get at least up to equal results. (All the scores are obtained by cross-validation on the very same test and training sets.)

Well, I thought the model space of the SVM and the RF should be more or less similar and therefore there should be similar models yielding similar results... — Peter Schneider
No there is no guarantee that the performance between the 2 models will be the same, the fact that you tried to optimise the params and performed scaling and normalisation does not guarantee that they should both perform as well. You can try the example on your dataset and see for yourself, you will ultimately find that there is no hard and fast rule or guarantee that any model will always perform best — EdChum
Did you normalize the columns of the feature matrix ? e.g . to range 0 .. 1 or mean 0 (plus std deviation of 1) ? — rocksportrocker

ogrisel ogrisel · Accepted Answer · 2014-09-08T16:48:25

As EdChum said in the comments there is no rule or guarantee that any model always perform best.

The SVM with RBF kernel model makes the assumption that the optimal decision boundary is smooth and rotation invariant (once you fix a specific feature scaling that is not rotation invariant).

The Random Forest does not make the smoothness assumption (it's a piece wise constant prediction function) and favors axis aligned decision boundaries.

The assumptions made by the RF model might just better fit the task.

BTW, thanks for having grid searched C and gamma and checked the impact of feature normalization before asking on stackoverflow :)

Edit to get some more insight, it might be interesting to plot the learning curves for the 2 models. It might be the case that the SVM model regularization and kernel bandwidth cannot deal with overfitting good enough while the ensemble nature of RF works best for this dataset size. The gap might get closer if you had more data. The learning curves plot is a good way to check how your model would benefit from more samples.

SVM poor performance compared to Random Forest

1 Answers