What is the classifier used in scikit-learn's VotingClassifier?

3
votes

I looked at the documentation of scikit-learn but it is not clear to me what sort of classification method is used under the hood of the VotingClassifier? Is it logistic regression, SVM or some sort of a tree method?

I'm interested in ways to vary the classifier method used under the hood. If Scikit-learn is not offering such an option is there a python package which can be integrated easily with scikit-learn which would offer such functionality?

EDIT:

I meant the classifier method used for the second level model. I'm perfectly aware that the first level classifiers can be any type of classifier supported by scikit-learn.

The second level classifier uses the predictions of the first level classifiers as inputs. So my question is - what method does this second level classifier use? Is it logistic regression? Or something else? Can I change it?

2
python-3.xscikit-learnensemble-learning
I answered your question below. Please let me know if my answer helped you. - Tim
@TimH Thanks but, unfortunately, your post is not answering my question. Please, see the edit. - Nick
What do you mean by second level classifier? - Tim

2 Answers

5
votes

General

The VotingClassifier is not limited to one specific method/algorithm. You can choose multiple different algorithms and combine them to one VotingClassifier. See example below: 

iris = datasets.load_iris()
X, y = iris.data[:, 1:3], iris.target

clf1 = LogisticRegression(...)
clf2 = RandomForestClassifier(...)
clf3 = SVC(...)

eclf = VotingClassifier(estimators=[('lr', clf1), ('rf', clf2), ('svm', clf3)], voting='hard')

Read more about the usage here: VotingClassifier-Usage. When it comes down to how the VotingClassifier "votes" you can either specify voting='hard' or voting='soft'. See the paragraph below for more detail.

Voting

Majority Class Labels (Majority/Hard Voting)

In majority voting, the predicted class label for a particular sample is the class label that represents the majority (mode) of the class labels predicted by each individual classifier.

E.g., if the prediction for a given sample is

classifier 1 -> class 1 classifier 2 -> class 1 classifier 3 -> class 2 the VotingClassifier (with voting='hard') would classify the sample as “class 1” based on the majority class label.

Source: scikit-learn-majority-class-labels-majority-hard-voting

Weighted Average Probabilities (Soft Voting)

In contrast to majority voting (hard voting), soft voting returns the class label as argmax of the sum of predicted probabilities.

Specific weights can be assigned to each classifier via the weights parameter. When weights are provided, the predicted class probabilities for each classifier are collected, multiplied by the classifier weight, and averaged. The final class label is then derived from the class label with the highest average probability.

Source/Read more here: scikit-learn-weighted-average-probabilities-soft-voting

2
votes

The VotingClassifier does not fit any meta model on the first level of classifiers output. It just aggregates the output of each classifier in the first level by the mode (if voting is hard) or averaging the probabilities (if the voting is soft).

In simple terms, VotingClassifier does not learn anything from the first level of classifiers. It only consolidates the output of individual classifiers.

If you want your meta model to be more intelligent, try using the adaboost, gradientBoosting models.