Replacing one layer in a Dense autoencoder with Conv1D

2
votes

I currently have a fully connected autoencoder in keras which looks like this:

model.add(Dense(4096, input_shape=(4096,), activation='relu'))
model.add(Dense(512, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(512, activation='relu'))
model.add(Dense(4096, activation='relu'))

The data consists of timerseries data which is converted into frequency domain using FFT.

My training data has the following shape: (8000, 4096) where i have 8000 samples and the 4096 samples represent the frequencies. This model is working fine.

What I'm trying to achieve is replacing the two dense layers which has 512 units with a Conv1d to see if this will improve my results, something like this:

model = Sequential()
model.add(Dense(4096, input_shape=(4096,), activation='relu'))
model.add(Conv1D(512,3, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Conv1D(512,3, activation='relu'))
model.add(Dense(4096, activation='relu'))

Now, this wont work because Conv1D is expecting my data to have 3 dimensions:

Input 0 is incompatible with layer conv1d_12: expected ndim=3, found ndim=2

How can I ensure that the Conv1d layers gets the correct input shape?

  1. Use Model.add(Reshape(?,?,?)) possibly?
  2. Reshape my input data to have 3 dimensions somehow?

I have tried changing the input shape to "force" a third dimension and reshaped between the first Dense layer and the first Conv1D layer, but that does not seem to work.

I realise that there are tons of questions regarding the input shape of Conv1D nets here, but please note that I do not want the convolutional filter to span across multiple samples, only across the frequency values.

Thanks in advance.

UPDATE: Following daniels advice I was able to compile the model and start training (allthough my GPU is screaming at me)

Layer (type)                 Output Shape              Param #   
=================================================================
dense_132 (Dense)            (None, 4096)              16781312  
_________________________________________________________________
reshape_85 (Reshape)         (None, 4096, 1)           0         
_________________________________________________________________
conv1d_71 (Conv1D)           (None, 4096, 512)         2048      
_________________________________________________________________
reshape_86 (Reshape)         (None, 2097152)           0         
_________________________________________________________________
dense_133 (Dense)            (None, 64)                134217792 
_________________________________________________________________
reshape_87 (Reshape)         (None, 64, 1)             0         
_________________________________________________________________
conv1d_72 (Conv1D)           (None, 64, 512)           2048      
_________________________________________________________________
reshape_88 (Reshape)         (None, 32768)             0         
_________________________________________________________________
dense_134 (Dense)            (None, 4096)              134221824 
=================================================================
Total params: 285,225,024
Trainable params: 285,225,024
Non-trainable params: 0
_________________________________________________________________

However, I would expect my Conv1d layers to have the following output shape:

conv1d_71 (Conv1D)           (None, 512, 1)

Am i doing the convolution over the wrong dimension? And if so, how can i change that?, or have i misunderstood how a convolutional layer works?

1
pythonmachine-learningkeras
Reshape your input to 3D: model.add(Reshape(input_shape=(4096,), target_shape=(4096, 1))) - Primusa
Thanks Primusa, but wont the cod you are showing here give a 2D output? e.g the target shape would have to be (4096,1,) or possibly (4096,1,1) ? Anyway, ill try and let you know - VegardKT
keras infers the first dimension as the batch size, you're really reshaping to (batch_size, 4096, 1) - Primusa
@Primusa Ah i see, then it makes sense. Where excactly would you add this method? I tried reshaping before the first conv1d, but then the Dense layer is throwin an dimensional error, and I dont seem the be allowed to change the shape after the first conv1d. (New size of array must be the same) - VegardKT
The first layer, that's why input_shape is specified. You might have to reshape it back into a 2D input during your output - Primusa

1 Answers

2
votes

Approach 1 - Keep length through dense layers

This won't be very useful as an autoencoder because it won't really compress your data, since the length of 4096 will still exist.

This should be working automatically, but it seems you've got an old keras version.

As you did, reshape it like (4096,1) before the first convolution.
Then the Dense layers should work ok. If not, use a TimeDistributed(Dense(...))

Consider using padding='same' in the convolutions for ease.

The last layer should then be something like a Conv1D(1,...) instead of a Dense. And finally a reshape with (4096,).

Approach 2 - Don't keep length

After the first conv, you should reshape it to remove the 3D dimensions, but notice that you have:

  • Channels: 512
  • Length:
    • If using padding='same': 4096
    • Else: 4094

Then you should reshape to (4096*512,), which is quite big and will significantly increase the amount of trainable params in your model.

Before the next conv, again to (64,1), then after it to (64*512,) (or 62 if not using padding).

You can explore a mix of these. Always remember that convs will work as (batch, length, channels) and denses will work as (batch, ...whatever..., units) if using recent keras versions. If not you must treate this ...whatever... properly.