How to input multidimensional data to caffe lmdb?

Question

I would like to reproduce the Two-Stream Convolutional Networks for Action Recognition in Videos.

But it feels Like I have Hit a wall when It comes to giving multi frame input to caffe.

As the single frame network gives 50% accuracy. But when I give an input of 30*227*227 via an LMDB. 20(10 frames each with 3 channels). The accuracy barely reaches 4%.

Which leads me to believe that the input I'm giving to caffe is not in the required format or model is wrong(less likely).

I would like to give an input in the style of 3 x 10 x 227 x 227. But the caffe LMDB has only a few variables as an input. Namely Height,widhth,channels,data,label.

Which lets me write data only in 3 dimensions and then cant the 4th frames dimension.

Does anyone know a work around this?

Or do u know Where I can find any examples of caffe using multi frame data for classification.

As far as I know Caffe uses 4D blobs to handle data and hence cant handle batches even If I get it to work. So it will work with only one sample at a time. And this also depends on how the filters are designed and If they can handle the correlation of the frames as they are in the 4th dimension which is generally used as a batch.

Any Ideas on this?

P.s I have also tried HDF5 similar results.

    datum = caffe.proto.caffe_pb2.Datum()
    datum.channels =3 #3 for a single image/ 30 for stacking them which I tried and get accuracy of 5%
    datum.height = 227
    datum.width = 227
    datum.data = X[i].tostring()  # or .tostring() if numpy < 1.9
    datum.label = classlabel
    str_id = '{:10}'.format(i+(counter))

Looking at your HDF5 results: the input size is 50x48x58x58. Is it related to this question? the accuracy starts with ~3% but then steadily rises... — Shai
@Shai Not really it was another experiment will try it again with HDF5 ... But I just went through the Two stream paper again and found that They actually give an input of w×h×2L where L is number of frames. But generally caffe accepts input of channelswh . And I'm not sure how they stacked the 2l as well for eg all Rs and then B and G. {R1,R2,... B1,B2...G1,G2....} or the {R1,B1,G1,R1,G2,B2.....} ... Any ideas ? — Arsenal Fanatic
I'm afraid I am not familiar with this work so I can't really answer. it depends on the way you wish to process the data — Shai

Dmitry Kostyaev Dmitry Kostyaev · Accepted Answer · 2016-05-30T15:20:09

According to the mentioned article, RGB image is an input for the Spatial stream and stacked optical flow frames for the Temporal stream. So, you should extract optical flow frames first for a temporal stream. Each point of the optical flow frame correspond to vector with x and y components that can be seen as image channels. By stacking L optical frames you will get an image with 2*L channels, and your input for caffe will be standard batch_sizex2LxHxW. Also take a look at this repo and code for more help.

How to input multidimensional data to caffe lmdb?

1 Answers