Data normalization Convolutional Autoencoders

Question

Iam a little bit confused about how to normalize/standarize image pixel values before training a convolutional autoencoder. The goal is to use the autoencoder for denoising, meaning that my traning images consists of noisy images and the original non-noisy images used as ground truth.

To my knowledge there are to options to pre-process the images: - normalization - standarization (z-score)

When normalizing using the MinMax approach (scaling between 0-1) the network works fine, but my question here is: - When using the min max values of the training set for scaling, should I use the min/max values of the noisy images or of the ground truth images?

The second thing I observed when training my autoencoder: - Using z-score standarization, the loss decreases for the two first epochs, after that it stops at about 0.030 and stays there (it gets stuck). Why is that? With normalization the loss decreases much more.

Thanks in advance,

cheers,

Mike

MinMax is really sensitive to noise and outliers, so I wouldn't use it in a denoising application. You can use quantiles 5% and 95% instead, or use z-score. For more realistic training, normalization should be performed on the noisy images — Pierre Gramme
And we don't have enough info to answer your second question. What loss? Any weight regularization used in the network? Please make it an independent question — Pierre Gramme
Thanks for your comment Per. Yes, z-score standarization seems to make sense. I using a MSE loss without any weight regularization. All layers use relu activation except for the last which uses Sigmoid. Cheers, Michael — Michael Lempart
Sigmoid will force its outputs between 0 and 1, so it is not suited for an autoencoder on z-score-transformed images (because target intensities can take arbitrary positive or negative values). — Pierre Gramme
Thanks Pierre. Do you think a linear activation would be better in this case? I also was thinking about my relu activations used in the hidden layer. If z score standardization is used, is relu a good choice then or would all negative values just be blocked? Thanks again, Michael — Michael Lempart

Pierre Gramme Pierre Gramme · Accepted Answer · 2019-10-22T16:51:58

[Note: This answer is a compilation of the comments above, for the record]

MinMax is really sensitive to outliers and to some types of noise, so it shouldn't be used it in a denoising application. You can use quantiles 5% and 95% instead, or use z-score (for which ready-made implementations are more common).

For more realistic training, normalization should be performed on the noisy images.

Because the last layer uses sigmoid activation (info from your comments), the network's outputs will be forced between 0 and 1. Hence it is not suited for an autoencoder on z-score-transformed images (because target intensities can take arbitrary positive or negative values). The identity activation (called linear in Keras) is the right choice in this case.

Note however that this remark on activation only concerns the output layer, any activation function can be used in the hidden layers. Rationale: negative values in the output can be obtained through negative weights multiplying the ReLU output of hidden layers.

Data normalization Convolutional Autoencoders

1 Answers