I'm fine-tuning caffenet on my own dataset. I've successfully done it on flickr style dataset, so I've modified solver and train_val prototxt files to point to my dataset, with new output dimension (=100 classes).
The size of my dataset is 1.88M images for training, and .48M images for testing.
When I started training, it paused for a while showing
... blocking_queue.cpp:50] Data layer prefetch queue empty
and then it proceeded as following:
... solver.cpp:414] Test net output #0: accuracy = 0.0002
... solver.cpp:414] Test net output #1: loss = 5.6139 (* 1 = 5.6139 loss)
... solver.cpp:242] Iteration 0, loss = 7.23329
... solver.cpp:258] Train net output #0: loss = 7.23329 (* 1 = 7.23329 loss)
... solver.cpp:571] Iteration 0, lr = 0.001
... solver.cpp:242] Iteration 20, loss = 0
... solver.cpp:258] Train net output #0: loss = 0 (* 1 = 0 loss)
... solver.cpp:571] Iteration 20, lr = 0.001
... solver.cpp:242] Iteration 40, loss = 0
... solver.cpp:258] Train net output #0: loss = 0 (* 1 = 0 loss)
... solver.cpp:571] Iteration 40, lr = 0.001
and so on. So apparently, no loss is being calculated at all. Weirdly, the initial loss at iteration 0 seems fair enough though.
So apparently, data were not loaded properly. I'm using text files to list the file paths and their labels.
I'm using softmax loss.
I've tried reducing initial learning rate to 1/10 but no change.
**edit: I've tried initial learning rate of 1/1000. Initially it seemed to work. After about 300 iterations, its loss quickly declined again and accuracy reached 100%, signifying that something might've gone wrong again. The symptoms appeared with both caffenet and VGG.
Any idea what might have possibly gone wrong?
