I have problem with object detection api, my loss is very low from start of the training:
INFO:tensorflow:global step 3: loss = 1.6555 (1.949 sec/step)
INFO:tensorflow:global step 4: loss = 1.1560 (2.021 sec/step)
INFO:tensorflow:global step 5: loss = 1.7363 (2.201 sec/step)
The mAP after few thousand steps is around 0.25... Does tensorflow save some additional "checkpoint" anywhere else instead training folder?? It seems like the model is using weights from previously trained network,even if have paths in config right to fresh checkpoint and everything is in different folders :(
I was following this blog tutorial [https://medium.com/towards-data-science/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9] on how to train a model in object detection api. I was using my own dataset with one class (just few images to test its working) and own config(faster rcnn resnet101). It did pretty well, model was able to detect class on new images. Then I try to train with 4 classes, a bit more images - worked nice.
But now I want to train on large set of 400 images, first I tried 4classes and problem with low loss appeared, so I tried with just one classes (I even labeled my dataset again to have only one class) same again. Now it is not working even with previous config on few images which I was training from begging.
Can someone help please?? Since now even the 5 images config which worked before is not working aswell, I dont think its related to bad dataset or something.. :(