High training accuracy, low validation accuracy CNN binary classification keras

Question

I'm trying to create a binary classifier that can differentiate between MRIs of alzheimer's patients and healthy individuals.
These are the stats so far:

• 1032 training images
• 400 validation images
• Running a simple model as shown below
• I have both the raw 160x160 images as well as the images after edge detection

Model:

model = Sequential([
    Conv2D(filters=16, kernel_size=(5, 5), activation='relu', padding = 'same', input_shape=(160,160,3)),
    MaxPool2D(pool_size=(2, 2), strides=2),
    Flatten(),
    Dense(units=2, activation='softmax')
])

As you can see - it's very simple, something I've done purposefully to try and remedy the issue of overfitting.

Output:

11/11 [==============================] - 2s 194ms/step - loss: 0.7604 - accuracy: 0.5155 - val_loss: 0.7081 - val_accuracy: 0.5000
Epoch 2/20
11/11 [==============================] - 2s 185ms/step - loss: 0.6885 - accuracy: 0.5223 - val_loss: 0.6942 - val_accuracy: 0.4839
Epoch 3/20
11/11 [==============================] - 2s 185ms/step - loss: 0.6802 - accuracy: 0.5854 - val_loss: 0.6985 - val_accuracy: 0.4931
Epoch 4/20
11/11 [==============================] - 2s 185ms/step - loss: 0.6717 - accuracy: 0.5932 - val_loss: 0.6996 - val_accuracy: 0.4677
Epoch 5/20
11/11 [==============================] - 2s 195ms/step - loss: 0.6512 - accuracy: 0.6175 - val_loss: 0.7124 - val_accuracy: 0.5115
Epoch 6/20
11/11 [==============================] - 2s 185ms/step - loss: 0.6345 - accuracy: 0.6476 - val_loss: 0.7073 - val_accuracy: 0.5253
Epoch 7/20
11/11 [==============================] - 2s 185ms/step - loss: 0.6118 - accuracy: 0.6680 - val_loss: 0.6920 - val_accuracy: 0.5207
Epoch 8/20
11/11 [==============================] - 2s 185ms/step - loss: 0.5817 - accuracy: 0.7068 - val_loss: 0.6964 - val_accuracy: 0.5207
Epoch 9/20
11/11 [==============================] - 2s 184ms/step - loss: 0.5528 - accuracy: 0.7272 - val_loss: 0.7123 - val_accuracy: 0.5161
Epoch 10/20
11/11 [==============================] - 2s 193ms/step - loss: 0.5239 - accuracy: 0.7417 - val_loss: 0.7397 - val_accuracy: 0.5392
Epoch 11/20
11/11 [==============================] - 2s 186ms/step - loss: 0.5106 - accuracy: 0.7427 - val_loss: 0.7551 - val_accuracy: 0.5461
Epoch 12/20
11/11 [==============================] - 2s 197ms/step - loss: 0.4920 - accuracy: 0.7650 - val_loss: 0.7402 - val_accuracy: 0.5438
Epoch 13/20
11/11 [==============================] - 2s 190ms/step - loss: 0.4741 - accuracy: 0.7835 - val_loss: 0.7564 - val_accuracy: 0.5507
Epoch 14/20
11/11 [==============================] - 2s 188ms/step - loss: 0.4591 - accuracy: 0.7767 - val_loss: 0.7445 - val_accuracy: 0.5300
Epoch 15/20
11/11 [==============================] - 2s 185ms/step - loss: 0.4486 - accuracy: 0.7767 - val_loss: 0.7712 - val_accuracy: 0.5415
Epoch 16/20
11/11 [==============================] - 2s 185ms/step - loss: 0.4503 - accuracy: 0.7806 - val_loss: 0.7446 - val_accuracy: 0.5346
Epoch 17/20
11/11 [==============================] - 2s 188ms/step - loss: 0.4404 - accuracy: 0.7670 - val_loss: 0.7669 - val_accuracy: 0.5553
Epoch 18/20
11/11 [==============================] - 2s 184ms/step - loss: 0.4169 - accuracy: 0.8078 - val_loss: 0.7804 - val_accuracy: 0.5576
Epoch 19/20
11/11 [==============================] - 2s 184ms/step - loss: 0.3987 - accuracy: 0.7971 - val_loss: 0.7846 - val_accuracy: 0.5507
Epoch 20/20
11/11 [==============================] - 2s 192ms/step - loss: 0.3977 - accuracy: 0.7981 - val_loss: 0.8060 - val_accuracy: 0.5461

Things I've tried so far:

• resizing the image to a smaller input
• adding dropout layers
• using preprocessed images where it's just the edges shown
• ensuring both classes in both training and validation datasets are evenly distributed
• changing learning rate
• reducing number of parameters to be of the same magnitude of the number of training images i have

I am literally out of ideas, I'm not sure how to move forward with this so I would appreciate any tips or advice.

All my code:

# Use ImageDataGenerator to create 3 lots of batches
train_batches = ImageDataGenerator(
    rescale=1/255).flow_from_directory(directory=train_path,
        target_size=(80,80), classes=['cn', 'ad'], batch_size=100,
            color_mode="rgb")
valid_batches = ImageDataGenerator(
    rescale=1/255).flow_from_directory(directory=valid_path,
        target_size=(80,80), classes=['cn', 'ad'], batch_size=100,
            color_mode="rgb")
# test_batches = ImageDataGenerator(
#     rescale=1/255).flow_from_directory(directory=test_path,
#         target_size=(224,224), classes=['cn', 'ad'], batch_size=10,
#             color_mode="rgb")

imgs, labels = next(train_batches)

# Test to see normalisation has occurred properly
print(imgs[1][8])

# Define method to plot MRIs
def plotImages(images_arr):
    fig, axes = plt.subplots(1, 10, figsize=(20,20))
    axes = axes.flatten()
    for img, ax in zip( images_arr, axes):
        ax.imshow(img)
        ax.axis('off')
    plt.tight_layout()
    plt.show()

# Plot a sample of MRIs
plotImages(imgs)

# # Define the model
# # VGG16
# model = Sequential()
# model.add(Conv2D(input_shape=(160,160,3),filters=64,kernel_size=(3,3),padding="same", activation="relu"))
# model.add(Conv2D(filters=64,kernel_size=(3,3),padding="same", activation="relu"))
# model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))
# model.add(Conv2D(filters=128, kernel_size=(3,3), padding="same", activation="relu"))
# model.add(Conv2D(filters=128, kernel_size=(3,3), padding="same", activation="relu"))
# model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))
# model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
# model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
# model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
# model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))
# model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
# model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
# model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
# model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))
# model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
# model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
# model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
# model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))
# model.add(Flatten())
# model.add(Dense(units=1024,activation="relu"))
# model.add(Dense(units=128,activation="relu"))
# model.add(Dense(units=2, activation="softmax"))

# # Model from the paper
# model = Sequential([
#     Conv2D(filters=32, kernel_size=(3, 3), activation='relu', padding = 'same', input_shape=(160,160,3)),
#     Conv2D(filters=32, kernel_size=(3, 3), activation='relu', padding='same'),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Flatten(),
#     Dense(units=2, activation='softmax')
# ])

## Model from Dr Paul
# static_conv_layer=Conv2D(filters=16, kernel_size=(5, 5), activation='relu', padding = 'same')
#
# model = Sequential([
#     Conv2D(filters=16, kernel_size=(5, 5), activation='relu', padding = 'same', input_shape=(32,32,3)),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.1),
#     static_conv_layer,
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.1),
#     Flatten(),
#     Dense(units=2, activation='softmax')
# ])

# This model hits around 75% train acc, 54% val acc
model = Sequential([
    Conv2D(filters=16, kernel_size=(5, 5), activation='relu', padding = 'same', input_shape=(80,80,3)),
    MaxPool2D(pool_size=(2, 2), strides=2),
    # Dropout(0.1),
    # Conv2D(filters=16, kernel_size=(3, 3), activation='relu', padding='same'),
    # MaxPool2D(pool_size=(2, 2), strides=2),
    # Conv2D(filters=16, kernel_size=(3, 3), activation='relu', padding='same'),
    # MaxPool2D(pool_size=(2, 2), strides=2),
    Flatten(),
    Dense(units=2, activation='softmax')
])

# model = Sequential([
#     Conv2D(filters=32, kernel_size=(3, 3), activation='relu', padding = 'same', input_shape=(160,160,3)),
#     Conv2D(filters=32, kernel_size=(3, 3), activation='relu', padding='same'),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Conv2D(filters=32, kernel_size=(3, 3), activation='relu', padding='same'),
#     Flatten(),
#     Dense(units=2, activation='softmax')
# ])

## Basic model with dropouts
# model = Sequential([
#     Conv2D(filters=32, kernel_size=(3, 3), activation='relu', padding = 'same', input_shape=(224,224,3)),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.1),
#     Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding='same'),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.2),
#     Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='same'),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.3),
#     Flatten(),
#     Dense(units=1, activation='sigmoid')
# ])

# Summarise each layer of the model
print(model.summary())

# Compile and train the model
model.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=['accuracy'])
model.fit(x=train_batches,
    steps_per_epoch=len(train_batches),
    validation_data=valid_batches,
    validation_steps=len(valid_batches),
    epochs=20,
    verbose=1
)

EDIT:

This paper seems to be doing much better than me and completing a very similar task, it may be useful to look at the methodology for:

Answer 1

Things you can try.

• It's a nice way to start with transfer learning. Using image net weights helps you to train just last layers and give much better accuracy.
• Adding early stopping and learning rate reduction with validation accuracy as constraint.
• Taking advantage of ImageDataGenerator and add much more data augmentation techniques.
• Make your model much deeper and also try different optimizer(RMSprop), run for more epochs with early stopping.
• Add callbacks and plot training validation accuracy graphs with respective to learning rate to see which lr proves best for the data.

Answer 2

It looks like your model is overfitting due to a lack of data. You can do some data augmentation to increase how many images you have. If you don't care about your aspect ratio you can warp the images, if you don't always need the full image you can crop it and you can rotate it if orientation is not important. These things can dramatically increase your dataset size and help mitigate overfitting.

Here is an example from the tensorflow documentation:

batch_size = 32
AUTOTUNE = tf.data.experimental.AUTOTUNE

def prepare(ds, shuffle=False, augment=False):
  # Resize and rescale all datasets
  ds = ds.map(lambda x, y: (resize_and_rescale(x), y), 
              num_parallel_calls=AUTOTUNE)

  if shuffle:
    ds = ds.shuffle(1000)

  # Batch all datasets
  ds = ds.batch(batch_size)

  # Use data augmentation only on the training set
  if augment:
    ds = ds.map(lambda x, y: (data_augmentation(x, training=True), y), 
                num_parallel_calls=AUTOTUNE)

  # Use buffered prefecting on all datasets
  return ds.prefetch(buffer_size=AUTOTUNE)

Also, here is a great video to watch from the TensorFlow developers youtube channel which explains the idea of image augmentation and shows an example of how to implement it.