Loss function for ordinal target on SoftMax over Logistic Regression

2
votes

I am using Pylearn2 OR Caffe to build a deep network. My target is ordered nominal. I am trying to find a proper loss function but cannot find any in Pylearn2 or Caffe.

I read a paper "Loss Functions for Preference Levels: Regression with Discrete Ordered Labels" . I get the general idea - but I am not sure I understand what will the thresholds be, if my final layer is a SoftMax over Logistic Regression (outputting probabilities).

Can some help me by pointing to any implementation of such a loss function ?

Thanks Regards

1
deep-learningtheanocaffelogistic-regressionsoftmax
you may need to describe your problem a little more if you still want some help. - user1269942
My label is ordinal, say 1,2,3,4,5. So I have 5 classes. My model is multi class using SoftMax over LogisticRegression,which is out putting 1,2,3,4,5. Now if the actual label for a trainings instance is 3 and my predicted output is 1 then that is worse than an predicted output of 2. Such a loss can be modeled if my model is regression using mean square error as the loss function. But in SoftMax, the loss is same. Meaning it can be equally worse to get a 2 against a 3 as compared to 1 against a 3 (using negative log likelihood). So how can I modify negative log likelihood to fit ordinal target? - Run2

1 Answers

0
votes

For both pylearn2 and caffe, your labels will need to be 0-4 instead of 1-5...it's just the way they work. The output layer will be 5 units, each is a essentially a logistic unit...and the softmax can be thought of as an adaptor that normalizes the final outputs. But "softmax" is commonly used as an output type. When training, the value of any individual unit is rarely ever exactly 0.0 or 1.0...it's always a distribution across your units - which log-loss can be calculated on. This loss is used to compare against the "perfect" case and the error is back-propped to update your network weights. Note that a raw output from PL2 or Caffe is not a specific digit 0,1,2,3, or 5...it's 5 number, each associated to the likelihood of each of the 5 classes. When classifying, one just takes the class with the highest value as the 'winner'.

I'll try to give an example... say I have a 3 class problem, I train a network with a 3 unit softmax. the first unit represents the first class, second the second and third, third.

Say I feed a test case through and get...

0.25, 0.5, 0.25 ...0.5 is the highest, so a classifier would say "2". this is the softmax output...it makes sure the sum of the output units is one.