with GTX 1050 ti, tensorflow gpu memory usage 100%, but load ~0

1
votes

I have an GTX 1050 ti (4GB) and i5 CPU, 8GB memory. I successfully installed tensorflow-gpu with cuda driver on win10 and the test shows that tensorflow is actually using the gpu (snapshot):enter image description here

However, when carrying out the training with CNN, while the GPU memory is always 100%, the GPU load is qualsi 0 with some spikes # 30%~70%:

Is it normal ?

enter image description here

EDIT: While the GPU occupation is qualsi 0 with spikes, the CPU load is fixed at 100% during the training.

EDIT2: I did read somewhere that the CPU could be high while GPU be low if there are a lot of operations of data copy between CPU and GPU. But I am using the official tensorflow object detection api for the training so I am totally unaware of the possible place in code.

1
tensorflow
Tensorflow will block the memory on the GPU for the python process so the memory consumption won't vary. Regarding the utilisation: GPU usage is very model and batch size dependent. Have a look here to make sure you input pipeline and data formats are optimised. As a starting tip: try increasing the batch size. – O. Gindele
@O.Gindele thank you. I tried the tensorflow object detection api, with different batch_size (froim 32 to 2). The result does not change much. The one info I did not methion is that the CPU occupation is fixed at 100% during the training. – captainst
Can you try increasing the batch size instead of lowering it? – O. Gindele
@O.Gindele Yeah I tried 32, 16, 8, 4, and 2. Pretty much the same result. I updated my original post with EDIT2. – captainst
@captainst How did installed tensorflow-gpu for 1050ti. I tried but it gave me an error. I am using cuda 10.0 toolkit, cudnn 7.4. The error I am getting is due to pywrap_tensorflow not found. – Aniket Bote

1 Answers

0
votes

What you see is normal behavior in most cases.

TensorFlow books the entire GPU memory initially.

The load on the GPU is dependent upon the data it is getting for processing.

If the data loading operation is slow, then most of the time GPU is waiting for data to get copied from disk to the GPU, and during that time it is not performing any work. That is what you see in your screen.