Caffe MemoryData Layer and Solver inteface

Question

I am trying to train a network but rather using lmdb or leveldb, I am feeding data to my network on the fly. So I am following procedure as outlined below

1. My data is loaded in Memory Data Layer.
2. I create a mini batch using a python script.
3. Set data and label as solver.net.set_input_arrays(batch,labels)
4. After that I call solver.step(1)

Here solver is of type SGDSolver. Now my question is what is the difference between solver.solve() and solver.step()?

2ndly this approach doesn't let me have a memory data layer for test network. Is there any work around for that?

My solver.prototxt looks like

net: "/path/to/train_val.prototxt"
base_lr: 0.01
lr_policy: "step"
gamma: 0.1
stepsize: 100000
display: 20
max_iter: 450000
momentum: 0.9
weight_decay: 0.0005
snapshot: 10000
snapshot_prefix: "/path/to/temporal_net_train"
solver_mode: GPU

With my approach every 20th iteration network displays some output loss etc. And somehow loss stays constant over some numbert of iterations, what could be the reason for that.

Answer 1

1. What is the difference between solver.solve and solver.step?

solve does the entire training run, to whatever limits you've set -- usually the iteration limit. step does only the specified number of iterations.

2. How do I get a memory data layer for test network?

If you're not reading from a supported data channel / format, I think you have to write a customer input routine (your data package).

3. Loss stays constant over multiple iterations, what could be the reason?

There are several possibilities, depending on the surrounding effects. If the loss shows only one value ever, then your back propagation is a likely culprit. Perhaps you're not properly connected to the data set, and you're not getting the expected classifications fed in.

If the loss has a temporary stability but then converges decently, don't worry about it; this is likely an effect of training ordering.

If the loss declines decently and then settles at a fixed value, then you're also doing well: the training converged before it ran out of iterations.