cudaGraphicsGLRegisterImage gives cudaErrorMemoryAllocation

Question

The Scenario

I'm creating two quite large, equally sized OpenGL 3D textures. The first one is a single-channel 16bit texture, the second one has four channels and 8 bits per texel. Than I need to register the two of them with CUDA.

The application I'm working on is 32bit. System characteristics: Win 7 64bit, NVIDIA GeForce 540M 2GB

The Problem

When using 512x512x391 or larger textures, my call to cudaGraphicsGLRegisterImage gives me a cudaErrorMemoryAllocation, although this return value is not even part of the list of possible return values, specified by the documentation of cudaGraphicsGLRegisterImage. One can find this list here on page 94. There are no memory-expansive operations performed before. However, it does work with smaller textures, such as 512x512x71.

Querying the free device memory right before the cudaGraphicsGLRegisterImage invocation tells about 1778 MB to be free.

Deliberations

1. One explanation might be that the return value is the error code of some previous invocation. I can basically eliminate this possibility by putting a cudaGetLastError right before the cudaGraphicsGLRegisterImage invocation and checking it's return value: it gives me a cudaSuccess.

2. I've performed some calculations. The memory required by the two 3D textures is:

   (512 * 512 * 391) texels * (4 channels * 1 bytes per texel + 1 channel * 2 bytes per texel) = 586.5 MB

   This is not that much, especially as my graphics device has 2 GB of dedicated memory.

3. I've also checked the total memory acquired by the process, just to make sure that this is none of this Win32-2GB-limitation issues: according to the task manager, the total process' memory size is about 279 MB, right before the cudaGraphicsGLRegisterImage invocation.

4. According to this posting, the memory costs of the cudaGraphicsRegisterImage operation should be very low.

Some Code

Acquisition of the two 3D textures:

glTexImage3D( GL_TEXTURE_3D, 0, GL_INTENSITY16
            , size.x, size.y, size.z
            , 0, GL_RED, GL_UNSIGNED_SHORT, bufferPtr );

glTexImage3D( GL_TEXTURE_3D, 0, GL_RGBA8
            , size.x, size.y, size.z
            , 0, GL_RGB, GL_BYTE, nullptr );

I've let off the texture generation and -binding for better overview.

Texture registration with CUDA:

size_t free, total;
CHECK_CUDA( cudaMemGetInfo( &free, &total ) );
qDebug() << "Free Memory:" << free / ( 1024 * 1024 ) << "MB";

enum resourceIndices{ FIRST_RESOURCE = 0, SECOND_RESOURCE = 1 };
cudaGraphicsResource* resources[ 2 ];

CHECK_CUDA( cudaGetLastError() );
CHECK_CUDA( cudaGraphicsGLRegisterImage( &resources[ FIRST_RESOURCE ]
                                       , firstTextureID
                                       , GL_TEXTURE_3D
                                       , cudaGraphicsRegisterFlagsReadOnly ) );

CHECK_CUDA( cudaGraphicsGLRegisterImage( &resources[ SECOND_RESOURCE ]
                                       , secondTextureID
                                       , GL_TEXTURE_3D
                                       , cudaGraphicsRegisterFlagsSurfaceLoadStore ) );

The macro CHECK_CUDA simply checks the return value and throws exceptions. It fails on the first cudaGraphicsGLRegisterImage invocation.

Answer 1

Your code snippets looks fine to me. However you said you're on a 32 bit system. And that could be the problem: Your process got only 2GiB of address space. And depending on how this address space was allocated, it may very well be, that there's no contiguous slice large enough left. So even if there's still plenty of memory, you simply may have run out of available address space. And since every piece of CUDA memory is mapped into process address space, even device memory, this may be very likely to happen.

It's a well known problem on 32 bit systems, known as address space fragmentation. This is actually the real benefit of 64 bit systems: Not so much the amount of memory addressable, but the sheer size of address space that simplifies memory management so much, because it is so hard to fill it up, that address space fragmentation is not a practical issue.

My suggestion: Test your program on a 64 bit system.