Activity regularizer with softmax?

Question

I have an l1 activity_regularizer=l1 in the final layer of my generative neural network:

outputs = Dense(200, activation='softmax', activity_regularizer=l1(1e-5))(x)

It makes my results better but I don't understand why it would change anything for a softmax activation. The sum of outputs = 1 , with all positive values always so the regularizer should give exactly the same loss no matter what.

What is the activity_regularizer=l1(1e-5) doing in my training ?

Answer 1

Due to the Softmax, the contribution of the L1-Regularization to the total cost is in fact constant.

However, the gradient of the regularization term is non-zero and equals to the number of non-zero activations (gradient of abs is sign, so we have a sum of signs of activations that are positive due to the softmax).

You can try to run with and without the L1-term and check how many non-zero elements you end up with.