understanding deconv layer math

Question

I need help. Trying to understand how the math of a deconv layer works. Let's talk about this layer:

layer {
  name: "decon"
  type: "Deconvolution"
  bottom: "conv2"
  top: "decon"
  convolution_param {
    num_output: 1
    kernel_size: 4
    stride: 2
    pad: 1
  }
}

So basically this layer is supposed to "upscale" an image by a factor of 2. If I look at the learned weights, I see e.g. this:

-0,0629104823 -0,1560362280 -0,1512266700 -0,0636162385
-0,0635886043 +0,2607241870 +0,2634004350 -0,0603787377
-0,0718072355 +0,3858278100 +0,3168329000 -0,0817491412
-0,0811873227 -0,0312164668 -0,0321144797 -0,0388795212

So far, so good. Now I'm trying to understand how to apply these weights to actually achieve the upscaling effect. I need to do this in my own code because I want to use simple pixel shaders.

Looking at the Caffe code, "DeconvolutionLayer::Forward_cpu" internally calls "backward_cpu_gemm", which does "gemm", followed by "col2im". My understanding of how all this works is this: gemm takes the input image, and multiplies each pixel with each of the 16 weights listed above. So basically gemm produces 16 output "images". Then col2im sums up these 16 "images" to produce the final output image. But due to the stride of 2, it stretches the 16 gemm images over the output image in such a way that each output pixel is only comprised of 4 gemm pixels. Does that sound correct to you so far?

My understand is that each output pixel is calculated from the nearest 4 low-res pixels, by using 4 weights from the 4x4 deconv weight matrix. If you look at the following image:

https://i.stack.imgur.com/X6iXE.png

Each output pixel uses either the yellow, pink, grey or white weights, but not the other weights. Do I understand that correctly? If so, I have a huge understanding problem, because in order for this whole concept to work correctly, e.g. the yellow weights should add up to the same sum as the pink weights etc. But they do not! As a result my pixel shader produces images where 1 out of 4 pixels is darker than the others, or every other line is darker, or things like that (depending on which trained model I'm using). Obviously, when running the model through Caffe, no such artifacts occur. So I must have a misunderstanding somewhere. But I can't find it... :-(

P.S: Just to complete the information: There's a conv layer in front of the deconv layer with "num_output" of e.g. 64. So the deconv layer actually has e.g. 64 4x4 weights, plus one bias, of course.

I don't fully understand what you mean my the colors, but here: arxiv.org/pdf/1603.07285v1.pdf you can find a well read about how the convolution arithmetics work. — sygi
Thanks. The PDF is interesting, but it didn't really help me much in solving my problem. In regards to the colors: If you look at Figure 4.6 in the PDF you linked, you can see that each "output pixel" is calculated by applying a 3x3 kernel on the "input image", which according to Figure 4.6 means each output pixel is created from either one or two input pixels. Now using a 3x3 kernel for a stride of 2 is rather weird. Caffe usually uses 4x4 for that. If you use a 4x4 kernel, each output pixel should be created by using 4 input pixels. That's the colored weights in the 4x4 weight matrix. — seeya

seeya seeya · Accepted Answer · 2016-11-12T11:30:39

After a lot of debugging I found that my understanding of the deconv layer was perfectly alright. I fixed all the artifacts by simply dividing the bias floats by 255.0. That's necessary because pixel shaders run in 0-1 range, while the Caffe bias constants seem to be targetted at 0-255 pixel values.

Everything working great now.

I still don't understand why the 4 weight pairs don't sum up to the same value and how that can possibly work. But what do I know. It does work, after all. I suppose some things will always be a mystery to me.

understanding deconv layer math

1 Answers