predict_proba or decision_function as estimator "confidence"

Question

I'm using LogisticRegression as a model to train an estimator in scikit-learn. The features I use are (mostly) categorical; and so are the labels. Therefore, I use a DictVectorizer and a LabelEncoder, respectively, to encode the values properly.

The training part is fairly straightforward, but I'm having problems with the test part. The simple thing to do is to use the "predict" method of the trained model and get the predicted label. However, for the processing I need to do afterwards, I need the probability for each possible label (class) for each particular instance. I decided to use the "predict_proba" method. However, I get different results for the same test instance, whether I use this method when the instance is by itself or accompanied by others.

Next, is a code that reproduces the problem.

from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction import DictVectorizer
from sklearn.preprocessing import LabelEncoder


X_real = [{'head': u'n\xe3o', 'dep_rel': u'ADVL'}, 
          {'head': u'v\xe3o', 'dep_rel': u'ACC'}, 
          {'head': u'empresa', 'dep_rel': u'SUBJ'}, 
          {'head': u'era', 'dep_rel': u'ACC'}, 
          {'head': u't\xeam', 'dep_rel': u'ACC'}, 
          {'head': u'import\xe2ncia', 'dep_rel': u'PIV'}, 
          {'head': u'balan\xe7o', 'dep_rel': u'SUBJ'}, 
          {'head': u'ocupam', 'dep_rel': u'ACC'}, 
          {'head': u'acesso', 'dep_rel': u'PRED'}, 
          {'head': u'elas', 'dep_rel': u'SUBJ'}, 
          {'head': u'assinaram', 'dep_rel': u'ACC'}, 
          {'head': u'agredido', 'dep_rel': u'SUBJ'}, 
          {'head': u'pol\xedcia', 'dep_rel': u'ADVL'}, 
          {'head': u'se', 'dep_rel': u'ACC'}] 
y_real = [u'AM-NEG', u'A1', u'A0', u'A1', u'A1', u'A1', u'A0', u'A1', u'AM-ADV', u'A0', u'A1', u'A0', u'A2', u'A1']

feat_encoder =  DictVectorizer()
feat_encoder.fit(X_real)

label_encoder = LabelEncoder()
label_encoder.fit(y_real)

model = LogisticRegression()
model.fit(feat_encoder.transform(X_real), label_encoder.transform(y_real))

print "Test 1..."
X_test1 = [{'head': u'governo', 'dep_rel': u'SUBJ'}]
X_test1_encoded = feat_encoder.transform(X_test1)
print "Features Encoded"
print X_test1_encoded
print "Shape"
print X_test1_encoded.shape
print "decision_function:"
print model.decision_function(X_test1_encoded)
print "predict_proba:"
print model.predict_proba(X_test1_encoded)

print "Test 2..."
X_test2 = [{'head': u'governo', 'dep_rel': u'SUBJ'}, 
           {'head': u'atrav\xe9s', 'dep_rel': u'ADVL'}, 
           {'head': u'configuram', 'dep_rel': u'ACC'}]

X_test2_encoded = feat_encoder.transform(X_test2)
print "Features Encoded"
print X_test2_encoded
print "Shape"
print X_test2_encoded.shape
print "decision_function:"
print model.decision_function(X_test2_encoded)
print "predict_proba:"
print model.predict_proba(X_test2_encoded)


print "Test 3..."
X_test3 = [{'head': u'governo', 'dep_rel': u'SUBJ'}, 
           {'head': u'atrav\xe9s', 'dep_rel': u'ADVL'}, 
           {'head': u'configuram', 'dep_rel': u'ACC'},
           {'head': u'configuram', 'dep_rel': u'ACC'},]

X_test3_encoded = feat_encoder.transform(X_test3)
print "Features Encoded"
print X_test3_encoded
print "Shape"
print X_test3_encoded.shape
print "decision_function:"
print model.decision_function(X_test3_encoded)
print "predict_proba:"
print model.predict_proba(X_test3_encoded)

Following is the output obtained:

Test 1...
Features Encoded
  (0, 4)    1.0
Shape
(1, 19)
decision_function:
[[ 0.55372615 -1.02949707 -1.75474347 -1.73324726 -1.75474347]]
predict_proba:
[[ 1.  1.  1.  1.  1.]]
Test 2...
Features Encoded
  (0, 4)    1.0
  (1, 1)    1.0
  (2, 0)    1.0
Shape
(3, 19)
decision_function:
[[ 0.55372615 -1.02949707 -1.75474347 -1.73324726 -1.75474347]
 [-1.07370197 -0.69103629 -0.89306092 -1.51402163 -0.89306092]
 [-1.55921001  1.11775556 -1.92080112 -1.90133404 -1.92080112]]
predict_proba:
[[ 0.59710757  0.19486904  0.26065002  0.32612646  0.26065002]
 [ 0.23950111  0.24715931  0.51348452  0.3916478   0.51348452]
 [ 0.16339132  0.55797165  0.22586546  0.28222574  0.22586546]]
Test 3...
Features Encoded
  (0, 4)    1.0
  (1, 1)    1.0
  (2, 0)    1.0
  (3, 0)    1.0
Shape
(4, 19)
decision_function:
[[ 0.55372615 -1.02949707 -1.75474347 -1.73324726 -1.75474347]
 [-1.07370197 -0.69103629 -0.89306092 -1.51402163 -0.89306092]
 [-1.55921001  1.11775556 -1.92080112 -1.90133404 -1.92080112]
 [-1.55921001  1.11775556 -1.92080112 -1.90133404 -1.92080112]]
predict_proba:
[[ 0.5132474   0.12507868  0.21262531  0.25434403  0.21262531]
 [ 0.20586462  0.15864173  0.4188751   0.30544372  0.4188751 ]
 [ 0.14044399  0.3581398   0.1842498   0.22010613  0.1842498 ]
 [ 0.14044399  0.3581398   0.1842498   0.22010613  0.1842498 ]]

As can be seen, the values obtained with "predict_proba" for the instance in "X_test1" change when that same instance is with others in X_test2. Also, "X_test3" just reproduces the "X_test2" and adds one more instance (that is equal to the last in "X_test2"), but the probability values for all of them change. Why does this happen? Also, I find it really strange that ALL the probabilities for "X_test1" are 1, shouldn't the sum of all be 1?

Now, if instead of using "predict_proba" I use "decision_function", I get the consistency in the values obtained that I need. The problem is that I get negative coefficients, and even some of the positives ones are greater than 1.

So, what should I use? Why do the values of "predict_proba" change that way? Am I not understanding correctly what those values mean?

Thanks in advance for any help you could give me.

UPDATE

As suggested, I changed the code so as to also print the encoded "X_test1", "X_test2" and "X_test3", as well as their shapes. This doesn't appear to be the problem, as the encoding is consistant for the same instances between the test sets.

Answer 1

As indicated on the question's comments, the error was caused by a bug in the implementation for the version of scikit-learn I was using. The problem was solved updating to the most recent stable version 0.12.1