Trouble Training SVM (scikit-learn package)

Question

Background / Question

I am trying to create a SVM using Scikit-learn. I have a training set (here is the link to it https://dl.dropboxusercontent.com/u/9876125/training_patients.txt) which I load and then use to train the SVM. The training set is 3600 lines long. When I use all 3600 tuples the SVM never finishes training.... BUT when I only use the first 3594 tuples it finishes training in under a minute. I've tried using a variety of different sized training sets and the same thing continues to happen... depending on how many tuples I use the SVM either trains very quickly or it never completes. This has led me to the conclusion that the SVM is having difficulty converging on an answer depeding on the data.

Is my assumption about this being a convergence problem correct? If so, what is the solution? If not, what other problem could it be?

Code

import pylab as pl  # @UnresolvedImport
from sklearn.datasets import load_svmlight_file

print(doc)
import numpy as np
from sklearn import svm, datasets


print "loading training setn"
X_train, y_train = load_svmlight_file("training_patients.txt")


h = .02  # step size in the mesh
C = 1.0  # SVM regularization parameter


print "creating svmn"
poly_svc = svm.SVC(kernel='poly', cache_size=600, degree=40, C=C).fit(X_train, y_train)


print "all done"

Try setting SVC to verbose=1. It doesn't tell you much, but it may give a hint as to whether it's still doing anything. (Also be aware that SVM training with SVC can take cubic time in the worst case.) — Fred Foo
As @FredFoo suggested, you can see if training of the model continues by setting verbose=1 and if it goes forever, you can limit maximum number of iterations by setting max_iter to a reasonable number. — baranbaris

lejlot lejlot · Accepted Answer · 2013-10-23T05:57:54

The optimization algorithm behind SVM has cubic (O(n^3)) complexity assuming relatively high cost (C) and high-dimensional feature space (polynomial kernel with d=40 implies ~1600 dimensional feature space). I would not call this "problems with convergence", as for over 3000 samples it can take a while to train such a model, and it is normal. The fact that for some subsets you achieve much faster convergence is the effect of very rich feature projection (the same can happen with RBF kernel) - and it is a common phenomenon, it is true even for very simple data from UCI library. As mentioned in the comments, setting "verbose=True" may give you additional information regarding your optimization process - it will output the number of iterations, the number of support vectors (higher the number of SVs, more is SVM overfitting, which can be also a reason for slow convergence).

Trouble Training SVM (scikit-learn package)

2 Answers