TensorFlow Object Detection API: Export process killed

Question

I have been working through the TensorFlow Object Detection guide to training an object detector on images of pets TensorFlow Link step-by-step, and now have trained model.ckpt files in /models/research directory. However, when I run export_inference_graph.py, the process is killed without error message and no frozen inference graph .pb file is created.

Run:

python object_detection/export_inference_graph.py \
     --input_type image_tensor \
     --pipeline_config_path object_detection/samples/configs/faster_rcnn_resnet101_pets.config \
     --trained_checkpoint_prefix model.ckpt-31394 \
     --output_directory exported_graphs

The last few lines:

>     ======================End of Report==========================
>     2019-03-04 22:32:13.368307: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:
> 2600000000 Hz
>     2019-03-04 22:32:13.370375: I tensorflow/compiler/xla/service/service.cc:150] XLA service
> 0x55ac8072fae0 executing computations on platform Host. Devices:
>     2019-03-04 22:32:13.370689: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor
> device (0): <undefined>, <undefined>
>     WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py:1266:
> checkpoint_exists (from tensorflow.python.training.checkpoi
>     nt_management) is deprecated and will be removed in a future version.
>     Instructions for updating:
>     Use standard file APIs to check for files with this prefix.
>     WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/tools/freeze_graph.py:232:
> convert_variables_to_constants (from tensorflow.python.f
>     ramework.graph_util_impl) is deprecated and will be removed in a future version.
>     Instructions for updating:
>     Use tf.compat.v1.graph_util.convert_variables_to_constants
>     WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/graph_util_impl.py:245:
> extract_sub_graph (from tensorflow.python.framewo
>     rk.graph_util_impl) is deprecated and will be removed in a future version.
>     Instructions for updating:
>     Use tf.compat.v1.graph_util.extract_sub_graph
>     Killed

Does anyone know what might be going wrong?

Eric McLachlan Eric McLachlan · Accepted Answer · 2019-11-12T09:29:57

I had a similar situation when running a TensorFlow Python application in a Docker container.

Situation:

The container starts fine, it's only when the classifiers models get loaded (at runtime) that the container starts to struggle. By default, my docker environment imposed a 1Gb upper limit on my Docker container's RAM allocation. Because this was not enough, the OS thrashed and eventually killed the container.

Solution:

Depending on your setup, you may be able to change the default RAM allocation of Docker via global settings (in my case, Docker Desktop for Windows). Otherwise, it's possible to start the container using docker run --memory=4g ..., which will allocate 4Gb of RAM to your container.

TensorFlow Object Detection API: Export process killed

1 Answers