Partially transparent window OpenGL/Win32

Question

I have read every existing question on this tricky (niche) subject, but I am stuck. I have a Win32 window with an OpenGL context. I want my window to be partially transparent.

My result so far is that the entirety of the window is transparent. I want only the black area to be transparent, so that I can draw some 3D objects and they will look like they are coming of the window.

First, in my window class, I have set hbrBackground to black.

Windows::WindowClass windowClass;
windowClass.cbSize = sizeof(Windows::WindowClass);
windowClass.style = Windows::CS_VREDRAW | Windows::CS_HREDRAW;
windowClass.lpfnWndProc = WndProc;
windowClass.cbClsExtra = 0;
windowClass.cbWndExtra = 0;
windowClass.hInstance = moduleHandle;
windowClass.hIcon = Windows::LoadIcon(0u, (X*)Windows::IDI_QUESTION);
windowClass.hCursor = Windows::LoadCursor(0u, (X*)Windows::IDC_ARROW);
windowClass.hbrBackground = Windows::CreateSolidBrush(0x00000000);
windowClass.lpszMenuName = nullptr;
windowClass.lpszClassName = (X*)name;
windowClass.hIconSm = Windows::LoadIcon(0u, (X*)Windows::IDI_QUESTION);

I have created my window with the WS_EX_LAYERED flag.

windowHandle = Windows::CreateWindow(Windows::WS_EX_LAYERED, (X*)name, "", Windows::WS_POPUP, w / 4, h / 4, w / 2, h / 2, 0u, 0u, moduleHandle, 0u);

In my pixel format, I have enabled alpha and composition.

PixelFormatDescriptor format;
format.nSize = sizeof(PixelFormatDescriptor);
format.nVersion = 1;
format.dwFlags = PFD_SUPPORT_OPENGL | PFD_DRAW_TO_WINDOW | PFD_DOUBLEBUFFER | PFD_SUPPORT_COMPOSITION;
format.iPixelType = PFD_TYPE_RGBA;
format.cColorBits = 32;
format.cRedBits = 0;
format.cRedShift = 0;
format.cGreenBits = 0;
format.cGreenShift = 0;
format.cBlueBits = 0;
format.cBlueShift = 0;
format.cAlphaBits = 8;
format.cAlphaShift = 0; 
format.cAccumBits = 0; 
format.cAccumRedBits = 0;
format.cAccumGreenBits = 0;
format.cAccumBlueBits = 0;
format.cAccumAlphaBits = 0;
format.cDepthBits = 24;
format.cStencilBits = 8; 
format.cAuxBuffers = 0; 
format.iLayerType = PFD_MAIN_PLANE;
format.bReserved = 0;
format.dwLayerMask = 0;
format.dwVisibleMask = 0; 
format.dwDamageMask = 0;

I have tried the blur region "trick", but it has no effect. My result is not related to this piece of code.

struct DwmBlurBehind
{
    U4 dwFlags;
    S4 fEnable;
    X* blurRegionHandle;
    S4 fTransitionOnMaximized;
};

DwmBlurBehind blurBehind;
blurBehind.dwFlags = Windows::DWM_BB_ENABLE | Windows::DWM_BB_BLURREGION;
blurBehind.fEnable = true;
blurBehind.blurRegionHandle = Windows::CreateRectRgn(0, 0, -1, -1);
blurBehind.fTransitionOnMaximized = false;
Windows::DwmEnableBlurBehindWindow(windowHandle, &blurBehind);

Finally, I have set the LWA_COLORKEY and LWA_ALPHA attributes. This is what gave me the effect displayed. However, the color key does not seem to be taken into account (I have tried non-zero values as well).

Windows::SetLayeredWindowAttributes(windowHandle, 0, 170, Windows::LWA_COLORKEY | Windows::LWA_ALPHA);

I did not forget to enable blending.

GL::Enable(GL::BLEND);
GL::BlendFunc(GL::SRC_ALPHA, GL::ONE_MINUS_SRC_ALPHA);

There is a high-performance technique that disables the redirection surface that is normally used when drawing into a window. It involves setting the WS_EX_NOREDIRECTION extended window style and calling CreateSwapChainForComposition() to create a swap chain that is presented directly to the windows composition module. You are using OpenGL but maybe you can adapt this method. — zett42
OpenGL blending is completely unrelated to translucency implemented by the windowing subsystem in Windows. Alpha blending in OpenGL ends at the OpenGL render context. — IInspectable

datenwolf datenwolf · Accepted Answer · 2018-01-30T18:33:42

What you want to do requires window compositing which has been around since Windows Vista, so essentially every version of Windows you have to care about (Windows XP and earlier are EoL).

The key steps to take is, to enable DWM intra window compositing by enabling "Blue Behind Window" and use a WM_POPUP window; if you do not use WM_POPUP style the window manager will draw decorations and your OpenGL rendering will "hover" above that.

        DWM_BLURBEHIND bb = {0};
        bb.dwFlags = DWM_BB_ENABLE;
        bb.fEnable = TRUE;
        bb.hRgnBlur = NULL;
        DwmEnableBlurBehindWindow(hWnd, &bb);

        MARGINS margins = {-1};
        impl_DwmExtendFrameIntoClientArea(hWnd, &margins);

Next you must create an OpenGL context using the newer "attribute" API instead of using the pixelformat descriptor selection. With the attribute API you can select a transparent with alpha window format.

int attribs[] = {
    WGL_DRAW_TO_WINDOW_ARB, TRUE,
    WGL_DOUBLE_BUFFER_ARB, TRUE,
    WGL_SUPPORT_OPENGL_ARB, TRUE, 
    WGL_PIXEL_TYPE_ARB, WGL_TYPE_RGBA_ARB,
    WGL_TRANSPARENT_ARB, TRUE,
    WGL_COLOR_BITS_ARB, 32,
    WGL_RED_BITS_ARB, 8,
    WGL_GREEN_BITS_ARB, 8,
    WGL_BLUE_BITS_ARB, 8,
    WGL_ALPHA_BITS_ARB, 8,
    WGL_DEPTH_BITS_ARB, 24,
    WGL_STENCIL_BITS_ARB, 8,
    0, 0
};

INT iPF;
UINT num_formats_choosen;
if( !wglChoosePixelFormatARB(
        hDC, 
        attribs, 
        NULL,
        1,
        &iPF,
        &num_formats_choosen) ) {
    fprintf(stderr, "error choosing proper pixel format\n");
    return NULL;
}
if( !num_formats_choosen ) {
    return NULL;
}

PIXELFORMATDESCRIPTOR pfd;
memset(&pfd, 0, sizeof(pfd));
/* now this is a kludge; we need to pass something in the PIXELFORMATDESCRIPTOR 
 * to SetPixelFormat; it will be ignored, mostly. OTOH we want to send something
 * sane, we're nice people after all - it doesn't hurt if this fails. */
DescribePixelFormat(hDC, iPF, sizeof(pfd), &pfd);

if( !SetPixelFormat(hDC, iPF, &pfd) ) {
    fprintf(stderr, "error setting proper pixel format\n");
    ReleaseDC(hWnd, hDC);
    DestroyWindow(hWnd);

    return NULL;
}

I've got a complete working example for this in my wglarb wrapper repository over at GitHub: https://github.com/datenwolf/wglarb/blob/master/test/layered.c

Partially transparent window OpenGL/Win32

3 Answers