TF2 Model: How run training on GPU, and evaluation on CPU

Question

I run training phase of TF2 model (based on object detection pre-trained models from TF2-Models Zoo) on GPU (Nvidia 3070). Is there some way to define evaluation phase (for checkpoints created by training) on CPU?

• Cause train phase allocates almost all memory of GPU, I cant run both of them (train and eval) on GPU.

• OS - Ubuntu 20.04

• GPU - Nvidia 3070 (driver 460)

• TF - 2.4.1

• Python - 3.8.5

Thank you.

Answer 1

In my case, the solution is into evaluation function define:

os.environ['CUDA_VISIBLE_DEVICES'] = '-1'