OpenGL Texture Mipmaps: Load and Unload

Question

In the perfect case, if the screen resolution is 1024x768, only 786432 texels are needed at a given moment, and 2 megabytes memory is enough. But in real world there is managing cost, so textures take much more memory.

Texture streaming could reduce the memory cost of texture, that is, not all the mipmaps of textures are needed at a given moment. A texture needs level 0 mipmap because it's near the camera, if it's far from the current camera, level 5 to 11 mipmaps may be enough. Camera moves in the scene a while, some mipmaps can be loaded and some mipmaps can be unloaded.

My question is how to do it effectively.

Say I have a 512x512 OpenGL texture in the scene, so it will has 10 mipmaps. From level 0 to level 9, there are: 512x512, 256x256, 128x128... and 1x1 mipmap. Simplely upload the data like this:

glBindTexture(GL_TEXTURE_2D, texId);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, 512, 512, 0, GL_RGBA, GL_UNSIGNED_BYTE, p1);
glTexImage2D(GL_TEXTURE_2D, 1, GL_RGBA8, 256, 256, 0, GL_RGBA, GL_UNSIGNED_BYTE, p2);
glTexImage2D(GL_TEXTURE_2D, 2, GL_RGBA8, 128, 128, 0, GL_RGBA, GL_UNSIGNED_BYTE, p3);
...
glBindTexture(GL_TEXTURE_2D, 0);

After a while, camera goes far from this texture in the scene, 64x64 mipmap is enough, so the top 3 mipmaps are going to be unloaded:

glBindTexture(GL_TEXTURE_2D, texId);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, 64, 64, 0, GL_RGBA, GL_UNSIGNED_BYTE, p4);
glTexImage2D(GL_TEXTURE_2D, 1, GL_RGBA8, 32, 32, 0, GL_RGBA, GL_UNSIGNED_BYTE, p5);
glTexImage2D(GL_TEXTURE_2D, 2, GL_RGBA8, 16, 16, 0, GL_RGBA, GL_UNSIGNED_BYTE, p6);
...
glBindTexture(GL_TEXTURE_2D, 0);

And then, camera moves towards this texture, 256x256 mipmap is needed:

glBindTexture(GL_TEXTURE_2D, texId);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, 256, 256, 0, GL_RGBA, GL_UNSIGNED_BYTE, p4);
glTexImage2D(GL_TEXTURE_2D, 1, GL_RGBA8, 128, 128, 0, GL_RGBA, GL_UNSIGNED_BYTE, p5);
glTexImage2D(GL_TEXTURE_2D, 2, GL_RGBA8, 64, 64, 0, GL_RGBA, GL_UNSIGNED_BYTE, p6);
...
glBindTexture(GL_TEXTURE_2D, 0);

This is just inefficient. Basically it recreates the texture everytime although the texture id is not changed. PBOs could makes it faster but the data copying is still cost.

For a 1024x1024 OpenGL texture, can I make it uses only the lower mipmaps, say level 1 to 9, and leave the level 0 mipmap empty (do not allocate in video memory) ? In other words: Always keep a subset of mipmaps from low level to high level. Load or unload higher level mipmaps without changing the lower mipmaps of a texture. I think in hardware's perspective this might be possible.

This is what I tried: If I don't call glTexImage2D for level 0 mipmap, this texture may be in an incomplete state. But if I call glTexImage2D with a null data pointer, it will be allocated in video memory with zero data (profiling via gDEBugger).

Answer 1

If your goal with all of this is limiting the amount of active texture memory you have available, nothing in OpenGL 4.5 is going to help you with that. Reallocation of the texture as a smaller one is a terrible idea (and no, the Unreal engine does not do that).

There are two cases where what you are talking about may matter. Case 1 would be what the Unreal engine uses it for: loading performance. It allocates the whole texture, all of its mipmaps, but it only loads the lower mipmaps first. This saves time on loading in the level. This can also be used to speed up streaming performance by doing the same thing.

This is done easily in OpenGL 4.5. That's what the mipmap range setting is for; you only load in the lower mipmaps and set the GL_TEXTURE_BASE_LEVEL and GL_TEXTURE_MAX_LEVEL to that range. OpenGL guarantees it will not try to access memory outside of that range.

But dynamically evicting mipmaps from memory is not something that OpenGL 4.5 has a mechanism for.

ARB_sparse_texture however does. It allows you to declare that certain mipmaps are "sparse". You allocate storage for them as normal, but you can declare that the higher levels may be evicted from memory. You decide when levels are available by giving them virtual pages. And you can remove these pages, which allows the GPU to use that memory for someone else.

The goal in doing this is, in theory, to be able to use more textures without running out of GPU memory (and thus thrashing). However, you don't really have the tools to be able to do this effectively.

This is because OpenGL doesn't tell you how much memory you have available. Nor does it say how much memory is allocated for buffers/textures. Nor does it say how many of those allocated buffers/textures is currently active and resident on the GPU. So you don't really know when you're close to being about to thrash.

This means that, if you're "close to" a number of large textures, you can still blow memory limits. And OpenGL will not tell you when you've done so.

Even so, this can still be helpful, if you plan your level layouts around it. Oh, and sparse textures are useful for the Unreal engine's case too.

---

I am thinking about that every mipmap of an OpenGL texture is stored independently, we can attach a lower level mipmap to a texture at runtime, why cannot attach a higher one.

Do not make the mistake of thinking that hardware follows what the API says.

Can you only issue glTexImage*D calls to mipmaps other than layer 0? Yes; just use the base/max levels to prevent access outside of the allocated range (that will keep the texture complete).

Will that guarantee that the implementation only allocates memory for those particular mipmap levels? No; implementations may allocate mipmap levels outside that range.

Indeed, there's evidence that it doesn't work that way. Consider ARB_texture_storage. This extension, core in OpenGL 4.3+, provides a single function that allocates all of the mipmap levels of the texture at once. So instead of calling glTexImage2D for each level, you call glTexStorage2D once, and it will allocate all specified mipmap levels for the size you specified. You can leave some off the small range, but not off the top.

This also makes the texture immutable, so you cannot change that texture's storage ever again. You can upload to it, but you cannot call glTexStorage*D or glTexImage*D on it. So no reallocation.

Why would the ARB create an extension who's whole purpose is to prevent you from allocating individual mipmaps, if individual allocations were something that hardware actually supported? And if you think that's a fluke, consider this too.

When ARB_direct_state_access was created, they obviously added DSA-style functions for texture allocation. But notice that they didn't add DSA-style functions to make non-immutable textures; they didn't add glTextureImage*D. Their reasoning? "Immutable texture is a more robust approach to handle textures".

Clearly, the ARB sees no value in an API that lets the user say which mipmaps are allocated and which are not.

And it should be noted that nothing in Direct3D 12, a much lower level API, allows this either. The answer to the question of how much memory a texture needs is purely a byte-count + alignment. It is not a series of byte counts, one-per-mipmap. And the functions to allocate resources similarly do not allow expanding mipmaps or anything of the like.

And I even checked Mantle's documentation. It has no facility to have an image's memory be non-contiguous. grBindObjectMemory (the function that associates memory storage with a texture) does not take a parameter specifying the mipmap level.

And for the sake of completeness, neither does Vulkan. It treats an images mipmap pyramid as a single, contiguous block of storage. Sparse images can work, but that works on page boundaries, not at the level of whole mipmaps.

So I would assume nothing about the nature of the hardware based on old OpenGL APIs.

Answer 2

so textures take much more memory.

Actually no. (EDIT) for a 1-dimensional texture the zeroth mipmap level consumes N bytes, the first N/2, the second N/4, and so on. So the total amount of bytes consumes is

sum(i in 0…){ N * 2^-i } = N * sum(i in 0…){2^-i}

That's a geometric series which converges to 2. So a mipmaped texture consumes exactly twice the memory of a not mipmaped texture. For a 2D texture it's 1/4th, 1/16th and so on. The larger the dimension of the texture the smaller is the mipmap overhead.

That's not "a lot".

A texture needs level 0 mipmap because it's near the camera, if it's far from the current camera, level 5 to 11 mipmaps may be enough.

There is no camera in OpenGL and that's not how the mipmaping level is determined. The Mipmap level being used is determined by the rate of change of the texture coordinate in screen coordinates (gradient of the texture coordinate).

After a while, camera goes far from this texture in the scene, 64x64 mipmap is enough

And maybe you're using the same texture for rendering on a primitive with a very steep texture coordinate gradient (low mipmap level) right after drawing it with a shallow texture coordinate gradient.

But in real world there is managing cost,

What I'm trying to say is, that the management overhead of trying to properly stream mipmaps is much higher and consumes much more GPU resources, than simply loading all the mipmaps and call it a day.

Also modern GPUs are capable of data fetches on their own; the graphics RAM is just a cache for the system RAM and when you load a texture it actually doesn't go to the graphics memory in the first place. The GPU will fetch the data it needs.