why is the first dense layer not "dense"?

Question

I created a simple topology with an input layer, and 2 dense hidden layers. There are 10 inputs and 10 neurons for each hidden layer.

I would have expected each hidden layer to have 10x10 weights/synapses, but it seems that the first hidden layer only has 1x10 weights:

    features = {"test_input" : tensorflow.range(10)}                                                                                                                                  
    feature_columns = [tensorflow.feature_column.numeric_column("test_input")]                                                                                                        
    input_layer = tensorflow.feature_column.input_layer(                                                                                                                              
        features=features,                                                                                                                                                            
        feature_columns=feature_columns)                                                                                                                                              

    hidden_layer_1 = tensorflow.layers.dense(                                                                                                                                         
        input_layer,                                                                                                                                                                  
        units=10,                                                                                                                                                                     
        activation=tensorflow.nn.relu,                                                                                                                                                
        name="hidden_layer_1")                                                                                                                                                        

    hidden_layer_2 = tensorflow.layers.dense(                                                                                                                                         
        hidden_layer_1,                                                                                                                                                               
        units=10,                                                                                                                                                                     
        activation=tensorflow.nn.relu,                                                                                                                                                
        name="hidden_layer_2")                                                                                                                                                        

    vars = tensorflow.trainable_variables()                                                                                                                                           

    print vars

It seems like the first hidden layer only connects each neuron to a single corresponding input, rather than densely connecting to each input for each neuron. Is this expected? Where is this behaviour documented?

vijay m vijay m · Accepted Answer · 2018-05-11T14:30:21

That is because the input to the dense layer should be of size [batch_size, feature_size]. And your input are [10, 1], thats why the weights are of size 1x10.

Just doing a tf.transpose(input_layer) to the input of the first dense layer would make the input [1, 10] and should give the desired size.

why is the first dense layer not "dense"?

1 Answers