Extracting features from caffe last layer

Question

I am trying to extract features from the last layer of a GoogleNet caffe model fine-tuned on car classification. Here's the deploy.prototxt. I tried a couple of things:

I took features from 'loss3_classifier_model' layer which is incorrect.
Now I am extracting features from 'pool5' layer in the model as given in prototxt.

I am not sure whether it's correct or not because the features I am extracting for different cars doesn't seem to have much difference. In other words, I am unable to differentiate cars using this last layer features, I used Euclidean distance on features (Is it correct?). I am not using using softmax as I don't want to classify them, I just want features and I rechecking them using euclidean distance.

These are the steps I followed:

## load the model 
net = caffe.Net('deploy.prototxt',
                caffe.TEST,
                weights ='googlenet_finetune_web_car_iter_10000.caffemodel') 

# resize the input size as I have only one image in my batch.
net.blobs["data"].reshape(1, 3, 224, 224)

# I read my image of size (x,y,3)
frame = cv2.imread(frame_path) 

bbox = frame[int(x1):int(x2), int(y1):int(y2)] # getting the car, # I have stored x1,x2,x3,x4 seperatly.
# resized my image to 224,224,3, network input size.
bbox = cv2.resize(bbox, (224, 224)) 

# to align my input to the input of the model 
bbox_input = bbox.swapaxes(1,2).reshape(3,224,224) 

# fed input image to the model.
net.blobs['data'].data[0] = bbox_input 
net.forward()

# features from pool5 layer or the last layer.
temp = net.blobs["pool5"].data[0]

Now, I want to confirm if these steps are correct or not? I am new to caffe and I am not sure about the steps I wrote above.

rafaspadilha rafaspadilha · Accepted Answer · 2018-03-10T05:41:12

Both options are valid. The farther you are from the end of your network, the less specialized the features will be to your problem/training set, while still capturing relevant information that may be applied to similar tasks. As you move to the end of the network, the features will be more tuned to your task.

Note that you are dealing with two similar problems/tasks. The network was fine-tuned for car classification ("which model is this car?") and now you want to verify if two cars belong to the same model.

Considering the network was fine-tuned with a large and representative training set, the features obtained from it are powerful and with a lot of representation capability (i.e., they capture a lot of complex underlying patterns of the task they were trained to) that are useful to your verification task.

With this in mind, you could try many ways of comparing two feature vectors:

Euclidean Distance is too simple. I would try it only because it is easy/fast to implement;
Cosine Similarity [1] might also be a simple, but good starting point;
Classifier. Another possibility that I've have done in a similar problem was to train a classifier (SVM, Logistic Regression) on top of a combination of the two features. The input of your classifier could be the concatenation of them side by side.
Incorporate the verification task to your network. You could alter the GoogleNet architecture to receive two photos of cars and output if they belong or not to the same model. You would be transforming/fine-tuning your network from the classification problem to a verification task. Check for siamese networks [2].

Edit: there is a mistake when you resize your frame that may be the cause of your problems!

# I read my image of size (x,y,3)
frame = cv2.imread(frame_path) 

# resized my image to 224,224,3, network input size.
bbox = cv2.resize(frame, (224, 224))

You should have passed frame as input in the cv2.resize() method. You are probably feeding a garbage input to the network and that is why the output ends up always looking similar.

Extracting features from caffe last layer

1 Answers