Fully Connected Layer dimensions

Question

I have a few uncertainties regarding the fully connected layer of a convolutional neural network. Lets say the the input is the output of a convolutional layer. I understand the previous layer is flattened. But can it have multiple channels? (for example, can the input to the fully connected layer be 16x16x3 (3 channels, flattened into a vector of 768 elements?)

Next, I understand the equation for outputs is,

outputs = activation(inputs * weights' + bias)

Is there 1 weight per input? (for example, in the example above, would there be 768 weights?)

Next, how many bias's are there? 1 per channel (so 3)? 1 no matter what? Something else?

Lastly, how do filters work in the fully connected layer? Can there be more than 1?

Answer 1

You might have a misunderstanding of how the fully connected neural network works. To get a better understanding of it, you could always check some good tutorials such as online courses from Stanford HERE

To answer your first question: yes, whatever dimensions you have, you need to flatten it before sending to fully connected layers.

To answer your second question, you have to understand that fully connected layer is actually a process of matrix multiplication followed by a vector addition:

input^T * weights + bias = output

where you have an input of dimension 1xIN, weights of size INxOUT, and output of size 1xOUT, so you have 1xIN * INxOUT = 1xOUT. Altogether, you will have INxOUT weights, and OUT weights for each input. You will also need OUT biases, such that the full equation is 1xIN * INxOUT + 1xOUT(bias term).

There is no filters since you are not doing convolution.

Note that fully connected layer is also equal to 1x1 convolution layer, and many implementations use later for fully connected layer, this could be confusing for beginners. For details, please refer to HERE