RuntimeError: CUDA out of memory. Tried to allocate… but memory is empty

Question

I'm trying to run the train file from this Unet with their default hyperparameters, batch size = 1.

I have a GTX970 with 4GB and made Windows use the integrated graphics.

When I run nvidia-smi, it says that the memory of the GPU is almost free (52MiB / 4096MiB), "No running processes found " and pytorch uses the GPU not the integrated graphics

I do not understand what is using the memory:

RuntimeError: CUDA out of memory. Tried to allocate 150.00 MiB (GPU 0; 4.00 GiB total capacity; 2.77 GiB already allocated; 72.46 MiB free; 2.82 GiB reserved in total by PyTorch).

Could you check if memory accumulates over time on the GPU? You can use nvidia-smi -l 1 to make the display loop every second. — Ivan
Seems like 4GB is not enough with your current hyper-parameters. Try reducing the input resolution of the images. — akshayk07
I tried to reduce the resolution but then to the point its is too small and unfortunately the problem persists — Vlad Peter
Tried with the "nvidia-smi -l 1" and found out the there is no other process that eats the memory(only the one that I ran) but the error persists (saying that 2.7GB is already allocated) — Vlad Peter

ihdv ihdv · Accepted Answer · 2021-01-16T05:33:45

GPU memory allocation is not done all at once. As the program loads the data and the model, GPU memory usage gradually increases until the training actually starts. In your case, the program has allocated 2.7GB and tries to get more memory before training starts, but there is not enough space. 4GB GPU memory is usually too small for CV deep learning algorithms.

RuntimeError: CUDA out of memory. Tried to allocate… but memory is empty

1 Answers