About Softmax function as output layer in preddictions

Question

I know the softmax activation function: The sum of the ouput layer with a softmax activation is equal to one always, that say: the output vector is normalized, also this is neccesary because the maximun accumalated probability can not exceeds one. Ok, this is clear.

But my question is the following: When the softmax is used as a classifier, is use the argmax function to get the index of the class. so, what is the difference between get a acumulative probability of one or higher if the important parameter is the index to get the correct class?

An example in python, where I made another softmax (really is not a softmax function) but the classifier works in the same way that the classifier with the real softmax function:

import numpy as np

classes = 10
classes_list = ['dog', 'cat', 'monkey', 'butterfly', 'donkey',
                'horse', 'human', 'car', 'table', 'bottle']

# This simulates and NN with her weights and the previous 
# layer with a ReLU activation
a = np.random.normal(0, 0.5, (classes,512)) # Output from previous layer
w = np.random.normal(0, 0.5, (512,1))       # weights
b = np.random.normal(0, 0.5, (classes,1))   # bias

# correct solution:
def softmax(a, w, b):
    a = np.maximum(a, 0) # ReLU simulation
    x = np.matmul(a, w) + b
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0), np.argsort(e_x.flatten())[::-1]

# approx solution (probability is upper than one):
def softmax_app(a, w, b):
    a = np.maximum(a, 0) # ReLU simulation
    w_exp = np.exp(w)
    coef = np.sum(w_exp)
    matmul = np.exp(np.matmul(a,w) + b)
    res = matmul / coef
    return res, np.argsort(res.flatten())[::-1]

teor = softmax(a, w, b)
approx = softmax_app(a, w, b)
class_teor = classes_list[teor[-1][0]]
class_approx = classes_list[approx[-1][0]]
print(np.array_equal(teor[-1], approx[-1]))
print(class_teor == class_approx)

The obtained class between both methods are always the same (I'm talking about preddictions, not to training). I ask this because I'm implementing the softmax in a FPGA device and with the second method it is not necessary 2 runs to calculate the softmax function: first to find the exponentiated matrix and the sum of it and second to perform the division.

ibarrond ibarrond · Accepted Answer · 2021-01-19T14:26:32

Let's review the uses of softmax:

You should use softmax if:
1. You are training a NN and want to limit the range of output values during training (you could use other activation functions instead). This can marginally help towards clipping the gradient.
2. You are performing inference on a NN and you want to obtain a metric on the "degree of confidence" of your classification result (in the range of 0-1).
3. You are performing inference on a NN and wish to get the top K results. In this case it is recommended as a way to have a "degree of confidence" metric to compare them.
4. You are performing inference on several NN (ensemble methods) and wish to average them out (otherwise their results wouldn't easily comparable).
You should not use (or remove) softmax if:
1. You are performing inference on a NN and you only care about the top class. Note that the NN could have been trained with Softmax (for better accuracy, faster convergence, etc..).

In your case, your insights are right: Softmax as an activation function in the last layer is meaningless if your problem only requires you to get the index of the maximum value during the inference phase. Besides, since you are targetting an FPGA implementation, this would only give you extra headaches.

About Softmax function as output layer in preddictions

1 Answers