Culling works with extracting planes from view-projection matrix but not with projection matrix

Question

I have implemented frustum culling by using the plane extraction method explained in this article.

The article mentions that if the matrix is a projection matrix, then the planes will be in view-space. So I need to transform my AABB coordinates to view space to do the culling tests. However, this doesn't work.
But if extract the planes from view-projection matrix and test with AABB coordinates in model space, everything works fine.

The only change I've done is update the frustum planes using view-projection matrix with every camera motion, and transform AABB coordinates to model space instead of view space.

Here is the relevant code. The lines commented with "diff" are the only changes between two versions.

Code for projection matrix based frustum culling:

// called only at initialization
void camera_set_proj_matrix(camera *c, mat4 *proj_matrix)
{
    c->proj_matrix = *proj_matrix;
    // diff
    extract_frustum_planes(&c->frustum_planes, &c->proj_matrix); 
}

void camera_update_view_matrix(camera *c)
{
    mat4_init_look(&c->view_matrix, &c->pos, &c->dir, &VEC3_UNIT_Y);
    mat4_mul(&c->vp_matrix, &c->proj_matrix, &c->view_matrix);
    // diff
}

void chunk_render(const chunk *c, chunk_pos pos, const camera *camera, GLuint mvp_matrix_location)
{
    mat4 model_matrix;
    block_pos bp = chunk_pos_to_block_pos(pos);
    mat4_init_translation(&model_matrix, bp.x, bp.y, bp.z);
    mat4 mv_matrix;
    mat4_mul(&mv_matrix, &camera->view_matrix, &model_matrix);

    vec4 min = {0, 0, 0, 1};
    vec4 max = {CHUNK_SIDE, CHUNK_HEIGHT, CHUNK_SIDE, 1};
    // diff: using model view matrix here
    mat4_mul_vec4(&min, &mv_matrix, &min); 
    // diff: using model view matrix here
    mat4_mul_vec4(&max, &mv_matrix, &max); 
    AABB aabb = {{min.x, min.y, min.z}, {max.x, max.y, max.z}};
    if (AABB_outside_frustum(&aabb, &camera->frustum_planes)) return;

    //draw
}

Looks like it's culling too much. Also note: This abnormal culling only happens when I'm looking at positive z direction.

Code for view-projection based culling:

void camera_set_proj_matrix(camera *c, mat4 *proj_matrix)
{
    c->proj_matrix = *proj_matrix; 
    // diff
}

void camera_update_view_matrix(camera *c)
{
    mat4_init_look(&c->view_matrix, &c->pos, &c->dir, &VEC3_UNIT_Y);
    mat4_mul(&c->vp_matrix, &c->proj_matrix, &c->view_matrix);
    // diff: update frustum planes based on view projection matrix now
    extract_frustum_planes(&c->frustum_planes, &c->vp_matrix);
}

void chunk_render(const chunk *c, chunk_pos pos, const camera *camera, GLuint mvp_matrix_location)
{
    mat4 model_matrix;
    block_pos bp = chunk_pos_to_block_pos(pos);
    mat4_init_translation(&model_matrix, bp.x, bp.y, bp.z);
    mat4 mv_matrix;
    mat4_mul(&mv_matrix, &camera->view_matrix, &model_matrix);

    vec4 min = {0, 0, 0, 1};
    vec4 max = {CHUNK_SIDE, CHUNK_HEIGHT, CHUNK_SIDE, 1};
    // diff: using model matrix now
    mat4_mul_vec4(&min, &model_matrix, &min); 
    // diff: using model matrix now
    mat4_mul_vec4(&max, &model_matrix, &max); 
    AABB aabb = {{min.x, min.y, min.z}, {max.x, max.y, max.z}};
    if (AABB_outside_frustum(&aabb, &camera->frustum_planes)) return;

    // draw
}

Perfect

I've no idea why projection only method would be working in a weird way when I'm transforming aabb appropriately :/

You are not transforming an AABB. You are transforming two corners of an AABB and rebuilding a new AABB from the result. — Frank
Which language is this supposed to be? Don't tag two different languages. — too honest for this site
@Frank ohhh, I hadn't thought about that.. The camera rotations would make the rebuilt aabb invalid.. Thanks, you can make that an answer. — wingerse
@toohonestforthissite It's C but should work equally for C++ too I guess. It's not a C specific question. — wingerse

Frank Frank · Accepted Answer · 2018-12-06T18:23:43

What does transforming an AABB actually mean?

Let's look at the problem in 2D. Say I have a 2D AABB (defined by a bottom-left and a top-right corner), and I want to rotate it by 45 degrees.

---------
|       |
|       |   ->   ???
|       |
---------

The actual region of space this represents would obviously look like a diamond:

    / \
  /     \
/         \
\         /
  \     /
    \ /

However, since we want to encode that as an AABB, the resulting AABB would have to look like this:

-------------
|    / \    |
|  /     \  |
|/         \|
|\         /|
|  \     /  |
|    \ /    |
-------------

However, looking at your code:

mat4_mul_vec4(&min, &model_matrix, &min); 
// diff: using model matrix now
mat4_mul_vec4(&max, &model_matrix, &max); 
AABB aabb = {{min.x, min.y, min.z}, {max.x, max.y, max.z}};

What you are doing is build an AABB who's BL and TR are the transformed BL and TR from the original AABB:

    / \    
  /     \  
/         \
-----------
\         /
  \     /  
    \ /

What you should be doing is transform all 8 corners of your original AABB and build a new AABB around that. But working with world-space culling planes is also absolutely fine for most cases.

Alternatively, if your problem suits itself well to bounding spheres, you can save yourself a lot of trouble by using that instead.

Culling works with extracting planes from view-projection matrix but not with projection matrix

1 Answers