one hot encoding of output labels

3
votes

While I understand the need to one hot encode features in the input data, how does one hot encoding of output labels actually help? The tensor flow MNIST tutorial encourages one hot encoding of output labels. The first assignment in CS231n(stanford) however does not suggest one hot encoding. What's the rationale behind choosing / not choosing to one hot encode output labels?

Edit: Not sure about the reason for the downvote, but just to elaborate more, I missed out mentioning the softmax function along with the cross entropy loss function, which is normally used in multinomial classification. Does it have something to do with the cross entropy loss function? Having said that, one can calculate the loss even without the output labels being one hot encoded.

1
machine-learningclassificationone-hot-encoding
I'm voting to close this question as off-topic because it is not about programming, as established in the help center. However, you may consider doing additional research on multi-class classification and logistic regression. – E_net4
This may help. – mattdns

1 Answers

2
votes

One hot vector is used in cases where output is not cardinal. Lets assume you encode your output as integer giving each label a number.

The integer values have a natural ordered relationship between each other and machine learning algorithms may be able to understand and harness this relationship, but your labels may be unrelated. There may be no similarity in your labels. For categorical variables where no such ordinal relationship exists, the integer encoding is not good.

In fact, using this encoding and allowing the model to assume a natural ordering between categories may result in unexpected results where model predictions are halfway between categories categories.

What a mean by that?

The idea is that if we train an ML algorithm - for example a neural network - it’s going to think that a cat (which is 1) is halfway between a dog and a bird, because they are 0 and 2 respectively. We don’t want that; it’s not true and it’s an extra thing for the algorithm to learn.

The same may happen when data is encoded in n dimensional space and vector has a continuous value. The result may be hard to interpret and map back to labels.

In this case, a one-hot encoding can be applied to label representation as it has clear interpretation and its values are separated each is in different dimension.

If you need more information or would like to see the reason for one-hot encoding for the perspective of loss function see https://www.linkedin.com/pulse/why-using-one-hot-encoding-classifier-training-adwin-jahn/