Opengl hook rendering does not render in view when hooked application draws to screen

Question

I have setup a dll injection to hook swapbuffers in gdi32.dll.

The injection works great and triggers my hook_swapbuffers() function.

I want the hook_swapbuffers to put an overlay (for now a red semi transparent quad) on top of the victim process - in this case an opengl based game.

hook_swapbuffers() calls the following:

// initialise gl states

glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
glClearDepth(1.0f);
glDisable(GL_DEPTH_TEST);
glDepthFunc(GL_ALWAYS);
glShadeModel(GL_SMOOTH);
glHint(GL_PERSPECTIVE_CORRECTION_HINT, GL_NICEST);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glEnable(GL_BLEND);
glDisable(GL_TEXTURE_2D);

// Draw

glClear( GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

glPushMatrix(); // store current matrix settings
glMatrixMode(GL_PROJECTION);
glLoadIdentity();

// set the limits of our ortho projection
glOrtho(-1.0, 1.0, -1.0, 1.0, 0.1, 100);

glBegin(GL_QUADS);

glColor4f(1.0f, 0.0f, 0.0f, 0.5f);     // Red

glVertex3f(1.0f, 1.0f, 1.0f);
glVertex3f(-1.0f, 1.0f, 1.0f);
glVertex3f(-1.0f, -1.0f, 1.0f);
glVertex3f(1.0f, -1.0f, 1.0f);

glEnd();

glPopMatrix(); // restore original matrix

// Cleanup gl states

glEnable(GL_DEPTH_TEST);
glDepthFunc(GL_LEQUAL);

It then calls the normal SwapBuffers function.

If I comment out my games draw calls, the hook draw works great and I get a red semi-transparent screen aligned quad rendered over my game (as all draws are commented out, the game is effectively a black canvas, now dark red with my quad overlayed).

Unfortunately the minute I put my games draws back in, my screen aligned quad no longer renders??? I can set the glClear to clear the colour buffer and that occurs correctly, so the hook is working and opengl is taking on the commands from the hook.

I'm thinking I have some rogue state from the game affecting the draws in the hook function. Maybe the viewport/camera or some bound data that doesn't need to exist for my hook test.

Any help or pointers will be greatly received.

@Rabbid76 you my friend are a legend! It was indeed the matrix stacks that hadn't been cleared to identity. Not sure if its good form on here, but If you post again as an answer, rather than comment I will mark it as such so you get the rep points :) — Hinchy

Rabbid76 Rabbid76 · Accepted Answer · 2018-05-29T09:25:32

In hook_swapbuffers the projection matrix stack is proper set, but the initialization of the model view matrix stack is missing.
So the model view matrix can have any state remaining from the previous rendering. This causes arbitrary results in the following.

Push init and pop the model view matrix stack as you do it with the projection matrix stack:

glClear( GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

glPushMatrix();
glMatrixMode(GL_PROJECTION);
glLoadIdentity();

// set the limits of our ortho projection
glOrtho(-1.0, 1.0, -1.0, 1.0, 0.1, 100);

glPushMatrix();
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();

....

// glMatrixMode(GL_MODELVIEW);
glPopMatrix();
glMatrixMode(GL_PROJECTION); 
glPopMatrix();

Note, in OpenGL there is one matrix stack for each matrix mode (See glMatrixMode). The matrix modes are GL_MODELVIEW, GL_PROJECTION, and GL_TEXTURE.
If you would draw a texture in the "hook", the you would have to initialize the texture matrix stack too.

Opengl hook rendering does not render in view when hooked application draws to screen

1 Answers