Object Detection API - eval ended before train (giving partial steps precision)

Question

I was training SSD + mobilenet using tensorflow object detection api on my 500~ images dataset (eval images = 10% of the dataset). I ran training and then eval processes using gcloud. The training process took 7 hours and the eval process took only 45 min. Running tensorflow showed that loss graph for all the steps (200K) but precision (from eval process) only up to 45K steps. Why would that happen? How can I measure the precision of my training session? running again the eval process will do the work?

Answer 1

You have to remove the max_eval line in your config proto. By default, there is a value of 10, so it will no further evaluate after 10 evaluations.