Customized convolutional layer in TensorFlow

Question

Let's assume i want to make the following layer in a neural network: Instead of having a square convolutional filter that moves over some image, I want the shape of the filter to be some other shape, say a rectangle, circle, triangle, etc (this is of course a silly example; the real case I have in mind is something different). How would I implement such a layer in TensorFlow?

I found that one can define custom layers in Keras by extending tf.keras.layers.Layer, but the documentation is quite limited without many examples. A python implementation of a convolutional layer by for example extending the tf.keras.layer.Layer would probably help as well, but it seems that the convolutional layers are implemented in C. Does this mean that I have to implement my custom layer in C to get any reasonable speed or would Python TensorFlow operations be enough?

Edit: Perhaps it is enough if I can just define a tensor of weights, but where I can customize entries in the tensor that are identically zero and some weights showing up in multiple places in this tensor, then I should be able to by hand build a convolutional layer and other layers. How would I do this, and also include these variables in training?

Edit2: Let me add some more clarifications. We can take the example of building a 5x5 convolutional layer with one output channel from scratch. If the input is say 10x10 (plus padding so output is also 10x10)), I would imagine doing this by creating a matrix of size 100x100. Then I would fill in the 25 weights in the correct locations in this matrix (so some entries are zero, and some entries are equal, ie all 25 weights will show up in many locations in this matrix). I then multiply the input with this matrix to get an output. So my question would be twofold: 1. How do I do this in TensorFlow? 2. Would this be very inefficient and is some other approach recommended (assuming that I want to later customize what this filter looks like and thus the standard conv2d is not good enough).

Edit3: It seems doable by using sparse tensors and assigning values via a previously defined tf.Variable. However I don't know if this approach will suffer from performance issues.

You could build quite efficient convolutional layer in PyTorch using it's unfold capability (I'm not aware of anything similar in Tensorflow unluckily), but I don't know whether that suits you. — Szymon Maszke
Thanks, maybe, but that might not be good enough for the real example I have in mind, but I will look into it. I am really looking for a way to customize as much as possible and preferably in tensorflow. But see my edit — Jonathan Lindgren
Some parts are not clear to me: but where I can customize entries in the tensor that are identically zero - what do you mean by where? Where as if where in the code? And what is identically zero? You just want to zero-out some weights in this tensor? Next part: some weights showing up in multiple places in this tensor - do you mean weights being shared or having the same values? What would the shape of those weights be (matrix, 3d tensors like filters in 2D convolution)? Clarification on those would be helpful (might help you in 12 hours or so if we are on the same page). — Szymon Maszke
Sorry for the confusion. I am imagining building a convolutional layer from scratch (let's say one channel for simplicity). In that case, if the filter is size say 5x5, there are only 25 different weights. However, if we write this operation as a matrix multiplication, these same weights show up in many places in the matrix, while many other elements in this matrix are zero (since only 25 entries in the matrix couple to one output). That is what I meant by those comments above. — Jonathan Lindgren
I now plan to do this by first defining say 25 weights by tf.Variable, then creating a tf.Tensor and put these weights into the correct entries, and other entries to zero, and then just use matmul to multiply the input with this matrix. I am wondering if this is recommended approach or if I am overlooking something. — Jonathan Lindgren

MichaelSB MichaelSB · Accepted Answer · 2019-04-10T22:38:51

Just use regular conv. layers with square filters, and zero out some values after each weight update:

   g = tf.get_default_graph()
   sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
   conv1_filter = g.get_tensor_by_name('conv1:0')
   sess.run(tf.assign(conv1_filter, tf.multiply(conv1_filter, my_mask)))

where my_mask is a binary tensor (of the same shape and type as your filters) that matches the desired pattern.

EDIT: if you're not familiar with tensorflow, you might get confused about using the code above. I recommend looking at this example, and specifically at the way the model is constructed (if you do it like this you can access first layer filters as 'conv1/weights'). Also, I recommend switching to PyTorch :)

Customized convolutional layer in TensorFlow

1 Answers