keras cnn model only predicts one class for all the test images

4
votes

I am trying to build an image classification model with 2 classes with (1) or without (0). I can build the model and get an accuracy of 1. which is too good to be true (which is an issue) but when I use predict_generator as I have my images in folders, it only returns 1 class 0 (without class). There seems to be an issue but I can't work it out, I have looked at a number of articles but I still can't fix the issue.

image_shape = (220, 525, 3) #height, width, channels
img_width = 96
img_height = 96
channels = 3

epochs = 10

no_train_images = 11957              #!ls ../data/train/* | wc -l
no_test_images = 652                 #!ls ../data/test/* | wc -l
no_valid_images = 6156               #!ls ../data/test/* | wc -l

train_dir = '../data/train/'
test_dir = '../data/test/'
valid_dir = '../data/valid/'


test folder structure is the following:
test/test_folder/images_from_both_classes.jpg


#!ls ../data/train/without/ | wc -l 5606        #theres no class inbalance
#!ls ../data/train/with/ | wc -l 6351

#!ls ../data/valid/without/ | wc -l 2899
#!ls ../data/valid/with/ | wc -l 3257

classification_model = Sequential()

# First layer with 2D convolution (32 filters, (3, 3) kernel size 3x3, input_shape=(img_width, img_height, channels))
classification_model.add(Conv2D(32, (3, 3), input_shape=input_shape))
# Activation Function = ReLu increases the non-linearity
classification_model.add(Activation('relu'))
# Max-Pooling layer with the size of the grid 2x2
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
# Randomly disconnets some nodes between this layer and the next 
classification_model.add(Dropout(0.2))

classification_model.add(Conv2D(32, (3, 3)))
classification_model.add(Activation('relu'))
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
classification_model.add(Dropout(0.2))

classification_model.add(Conv2D(64, (3, 3)))
classification_model.add(Activation('relu'))
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
classification_model.add(Dropout(0.25))

classification_model.add(Conv2D(64, (3, 3)))
classification_model.add(Activation('relu'))
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
classification_model.add(Dropout(0.3))

classification_model.add(Flatten())
classification_model.add(Dense(64))
classification_model.add(Activation('relu'))
classification_model.add(Dropout(0.5))
classification_model.add(Dense(1))
classification_model.add(Activation('sigmoid'))

# Using binary_crossentropy as we only have 2 classes
classification_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])



batch_size = 32

# this is the augmentation configuration we will use for training
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    zoom_range=0.2)

# this is the augmentation configuration we will use for testing:
# only rescaling
valid_datagen = ImageDataGenerator(rescale=1. / 255)
test_datagen = ImageDataGenerator()

train_generator = train_datagen.flow_from_directory(
    train_dir,
    target_size = (img_width, img_height),
    batch_size = batch_size,
    class_mode = 'binary',
    shuffle = True)

valid_generator = valid_datagen.flow_from_directory(
    valid_dir,
    target_size = (img_width, img_height),
    batch_size = batch_size,
    class_mode = 'binary',
    shuffle = False)

test_generator = test_datagen.flow_from_directory(
    test_dir,
    target_size = (img_width, img_height),
    batch_size = 1,
    class_mode = None,
    shuffle = False)

mpd = classification_model.fit_generator(
    train_generator,
    steps_per_epoch = no_train_images // batch_size,         # number of images per epoch
    epochs = epochs,                                         # number of iterations over the entire data
    validation_data = valid_generator,
    validation_steps = no_valid_images // batch_size)

Epoch 1/10 373/373 [==============================] - 119s 320ms/step - loss: 0.5214 - acc: 0.7357 - val_loss: 0.2720 - val_acc: 0.8758

Epoch 2/10 373/373 [==============================] - 120s 322ms/step - loss: 0.2485 - acc: 0.8935 - val_loss: 0.0568 - val_acc: 0.9829

Epoch 3/10 373/373 [==============================] - 130s 350ms/step - loss: 0.1427 - acc: 0.9435 - val_loss: 0.0410 - val_acc: 0.9796

Epoch 4/10 373/373 [==============================] - 127s 341ms/step - loss: 0.1053 - acc: 0.9623 - val_loss: 0.0197 - val_acc: 0.9971

Epoch 5/10 373/373 [==============================] - 126s 337ms/step - loss: 0.0817 - acc: 0.9682 - val_loss: 0.0136 - val_acc: 0.9948

Epoch 6/10 373/373 [==============================] - 123s 329ms/step - loss: 0.0665 - acc: 0.9754 - val_loss: 0.0116 - val_acc: 0.9985

Epoch 7/10 373/373 [==============================] - 140s 376ms/step - loss: 0.0518 - acc: 0.9817 - val_loss: 0.0035 - val_acc: 0.9997

Epoch 8/10 373/373 [==============================] - 144s 386ms/step - loss: 0.0539 - acc: 0.9832 - val_loss: 8.9459e-04 - val_acc: 1.0000

Epoch 9/10 373/373 [==============================] - 122s 327ms/step - loss: 0.0434 - acc: 0.9850 - val_loss: 0.0023 - val_acc: 0.9997

Epoch 10/10 373/373 [==============================] - 125s 336ms/step - loss: 0.0513 - acc: 0.9844 - val_loss: 0.0014 - val_acc: 1.0000

valid_generator.batch_size=1
score = classification_model.evaluate_generator(valid_generator, 
                                                no_test_images/batch_size, pickle_safe=False)
test_generator.reset()
scores=classification_model.predict_generator(test_generator, len(test_generator))

print("Loss: ", score[0], "Accuracy: ", score[1])

predicted_class_indices=np.argmax(scores,axis=1)
print(predicted_class_indices)

labels = (train_generator.class_indices)
labelss = dict((v,k) for k,v in labels.items())
predictions = [labelss[k] for k in predicted_class_indices]

filenames=test_generator.filenames
results=pd.DataFrame({"Filename":filenames,
                      "Predictions":predictions})

print(results)

Loss: 5.404246180551993e-06 Accuracy: 1.0

print(predicted_class_indices) - ALL 0

[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

                              Filename Predictions
0      test_folder/video_3_frame10.jpg   without
1    test_folder/video_3_frame1001.jpg   without
2    test_folder/video_3_frame1006.jpg   without
3    test_folder/video_3_frame1008.jpg   without
4    test_folder/video_3_frame1009.jpg   without
5    test_folder/video_3_frame1010.jpg   without
6    test_folder/video_3_frame1013.jpg   without
7    test_folder/video_3_frame1014.jpg   without
8    test_folder/video_3_frame1022.jpg   without
9    test_folder/video_3_frame1023.jpg   without
10    test_folder/video_3_frame103.jpg   without
11   test_folder/video_3_frame1036.jpg   without
12   test_folder/video_3_frame1039.jpg   without
13    test_folder/video_3_frame104.jpg   without
14   test_folder/video_3_frame1042.jpg   without
15   test_folder/video_3_frame1043.jpg   without
16   test_folder/video_3_frame1048.jpg   without
17    test_folder/video_3_frame105.jpg   without
18   test_folder/video_3_frame1051.jpg   without
19   test_folder/video_3_frame1052.jpg   without
20   test_folder/video_3_frame1054.jpg   without
21   test_folder/video_3_frame1055.jpg   without
22   test_folder/video_3_frame1057.jpg   without
23   test_folder/video_3_frame1059.jpg   without
24   test_folder/video_3_frame1060.jpg   without

...just some of the outputs but all 650+ are without class.

This is the output and as you can see all the predicted values are 0 for the without class.

This is my first attempt at using Keras and CNN so any help would be really appreciated.

UPDATE

I have solved this. I am currently working on the accuracy but the main problem is now solved.

This is the line that caused problems.

predicted_class_indices=np.argmax(scores,axis=1)

argmax would be returning the index position of the result but as I was using binary classes and in my final layer I had 1 dense. It will only return a single value so it will always return the first class (0 as the index position). As the network is only set, to return one class.

Changing the following fixed my issue.

  1. Changed the class_mode to 'categorical' for the train and test generators
  2. Changed the final dense layer from 1 to 2 so this will return scores/probabilities for both classes. So when you use argmax, it will return the index position of the top score indicating which class it has predicted.
2
pythonkerasconv-neural-network
Are you saying that "with" and "without" have been switched? – Moondra
As for you test generator,it seems you forgot to normalize/preproces (1/255). – Moondra
no, it always predicts without class 0. I tried your suggestion and below which is the same but I still have the same issue. – vis7
Well there is a problem with the test generator. It could be that data is radically different from you train/valid which I doubt. You say you added the preprocess element to the generator (1/255) which was the other big thing. I am curious what does len(test_generator) produce if you do it standalone? I usuallly use test_generator.filenames. I doubt it will make a difference, but at this point I am not sure. – Moondra
One more thing you could do is check the valid_generator, with "predict_generator" and not "evaluate_generator" to make sure predict_generator isnt the problem. – Moondra

2 Answers

0
votes

You should change this line :

test_datagen = ImageDataGenerator()

By :

test_datagen = ImageDataGenerator(rescale=1. / 255)

If you do not pre-process your test set in the same way than your train/valid set, you will not get the expected results

0
votes

Give it some more time by try using some more epochs (e.g. 50). Also change the learning rate (divide it by 10 each time you try) and other regularization parameters.