Keras VGG16 modified model giving the same prediction every time

Question

I have taken the inbuilt keras.applications.vgg16.VGG16(weights='imagenet', include_top=True,input_shape=(224,224,3)) model and did transfer learning for PASCAL VOC 2012 dataset with 20 classes by including Global Average Pooling Layer as given below:

def VGG16_modified():
    base_model = vgg16.VGG16(include_top=True,weights='imagenet',input_shape=(224,224,3))
    print(base_model.summary())
    x = base_model.get_layer('block5_pool').output
    x = (GlobalAveragePooling2D())(x)
    predictions = Dense(20,activation='sigmoid')(x)

    final_model = Model(input = base_model.input, output = predictions)
    print(final_model.get_weights())
    return final_model

Now, I want to take the Class Activation Map based on this paper. For this, my code is as given below:

 def get_CAM(model,img):
        model = load_model(model)
        im = image.load_img(img,target_size=(224,224))
        im = image.img_to_array(im)
        im = np.expand_dims(im,axis=0)
        class_weights = model.layers[-1].get_weights()[0]
        final_conv_layer = model.get_layer('block5_pool')
        cam_model = Model(inputs = model.input,outputs=(final_conv_layer.output,model.layers[-1].output))
        conv_outputs, predictions = cam_model.predict(im)
        conv_outputs = np.squeeze(conv_outputs)
        prediction = np.argmax(predictions)
        print(predictions)
        print(prediction)
        print(conv_outputs)
        print(conv_outputs.shape)
        class_weights = class_weights[:,prediction]
        mat_for_mult = scipy.ndimage.zoom(conv_outputs,(32,32,1),order=1)
        final_output = np.dot(mat_for_mult.reshape((224*224, 512)),class_weights).reshape((224,224))
        print(final_output)
        return final_output

But cam_model.predict(im) is always giving the same class for all images. I am not sure where have I wrong with this. As the pascal voc 2012 contains multi label images, I have used 'sigmoid' in the last layer of the modified_vgg16 rather than 'softmax'. Can you let me know where have I gone wrong.

PASCAL VOC has multi label images. And I noticed that around 50% images have 'Person' class label and now all the images are getting classified to 'Person'. Just wondering if it is the issue. If so, how do we resolve this. — Sree
Since you are using sigmoid, each class is supposed to be independent. I notice you are using np.argmax(predictions), it would make sense if apply softmax on last layer. However, for multi-label tasks, usually, I will predict multiple classes based on the threshold of each class. — zihaozhihao
@zihaozhihao, yeah I have seen that, I thought of changing to threshold value, but I am struck with this issue. The 'Person' class is always getting highest probability here. — Sree
Well, the imbalanced issue could be alleviated during the training, such as introducing the class weights. — zihaozhihao

karandeep36 karandeep36 · Accepted Answer · 2019-10-15T09:01:04

PASCAL VOC has imbalanced distribution of images between classses.

Most of the images in dataset are for person which leads to imbalance data distribution and creates bias towards person class.

In order to avoid it, you can use the method of weighting the classes which helps to remove imbalance between classes

Please refer to this for more information on how to set class weights for imbalanced class.

Keras VGG16 modified model giving the same prediction every time

1 Answers