I am running a training phase on caffe. The network is: VGG16 + 8 convolution layers + softmax loss. Because of my machine, I choose to run on CPU only. My memory is 8G and my dataset is VOC2007 segmentation images(train+val+test no more than 1G memory).
However, after entering the command, the program stick at a point with message "Data layer prefetch queue empty" for long long time(~10 hours). During the time, the system monitor shows my CPU is 100% used and Memory 69%. Since my train batch is only 10, the number I think my machine can afford, it's hardly to understand why this happened?
Below is the output message after Network initialization done and Solver scaffolding done.
I0404 13:08:15.401798 28175 net.cpp:274] Network initialization done.
I0404 13:08:15.401984 28175 solver.cpp:60] Solver scaffolding done.
I0404 13:08:15.402128 28175 caffe.cpp:129] Finetuning from /home/huanghe/caffe-dilation/pretrained/frontend_model.caffemodel
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 537388947
I0404 13:08:22.585640 28175 net.cpp:752] Ignoring source layer prob
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 537388947
I0404 13:08:23.729429 28175 net.cpp:752] Ignoring source layer prob
I0404 13:08:23.732425 28175 caffe.cpp:219] Starting Optimization
I0404 13:08:23.732533 28175 solver.cpp:279] Solving
I0404 13:08:23.732580 28175 solver.cpp:280] Learning Rate Policy: step
I0404 13:08:23.833487 28175 solver.cpp:337] Iteration 0, Testing net (#0)
I0404 13:08:23.891206 28175 blocking_queue.cpp:50] Data layer prefetch queue empty
I am a new hand. The search on internet never helps me out. Can anyone explain it for me? Thank you very much!
