Program does work correctly from TCL exec

Question

I have the following C++ program written/borrowed that takes a screenshot of a window based on the window name.

When I run the program through a Windows command prompt, it works correctly. However, when I call the program in a TCL script with the exec command, it crashes the wish86 application.

Why does the program work through the command line, but not with the exec command?

Example: screenshot.exe calc.bmp "Calculator"

#include <windows.h>
#include <gdiplus.h>
#include <stdio.h>

#pragma comment(lib, "user32.lib")
#pragma comment(lib, "gdi32.lib")
#pragma comment(lib, "gdiplus.lib")

using namespace Gdiplus;

// From http://msdn.microsoft.com/en-us/library/ms533843%28VS.85%29.aspx
int GetEncoderClsid(const WCHAR* format, CLSID* pClsid)
{
    UINT  num = 0;          // number of image encoders
    UINT  size = 0;         // size of the image encoder array in bytes

    ImageCodecInfo* pImageCodecInfo = NULL;

    GetImageEncodersSize(&num, &size);
    if(size == 0)
        return -1;  // Failure

    pImageCodecInfo = (ImageCodecInfo*)(malloc(size));
    if(pImageCodecInfo == NULL)
        return -1;  // Failure

    GetImageEncoders(num, size, pImageCodecInfo);

    for(UINT j = 0; j < num; ++j) {

        if( wcscmp(pImageCodecInfo[j].MimeType, format) == 0 ) {
            *pClsid = pImageCodecInfo[j].Clsid;
            free(pImageCodecInfo);
            return j;  // Success
        }
    }

    free(pImageCodecInfo);
    return -1;  // Failure
}

int wmain(int argc, wchar_t** argv)
{

    GdiplusStartupInput gdiplusStartupInput;
    ULONG_PTR gdiplusToken;
    GdiplusStartup(&gdiplusToken, &gdiplusStartupInput, NULL);

    HDC desktopdc;
    HDC mydc;
    HBITMAP mybmp;

    desktopdc = GetDC(NULL);

    if(desktopdc == NULL) {
        return -1;
    }

    // If three arguments were passed, capture the specified window
    if(argc == 3) {

        RECT rc;

        // Convert wchar_t[] to char[]
        char title[512] = {0};

        wcstombs(title, argv[2], wcslen(argv[2]));

        HWND hwnd = FindWindow(NULL, title);    //the window can't be min

        if(hwnd == NULL) {
            return -1;
        }

        GetWindowRect(hwnd, &rc);
        mydc = CreateCompatibleDC(desktopdc);

        mybmp = CreateCompatibleBitmap(desktopdc, rc.right - rc.left, rc.bottom - rc.top);
        SelectObject(mydc,mybmp);

        //Print to memory hdc
        PrintWindow(hwnd, mydc, PW_CLIENTONLY);

    } else {

        // Capture the entire screen
        int width = GetSystemMetrics(SM_CXVIRTUALSCREEN);
        int height = GetSystemMetrics(SM_CYVIRTUALSCREEN);

        mydc = CreateCompatibleDC(desktopdc);

        mybmp = CreateCompatibleBitmap(desktopdc, width, height);
        HBITMAP oldbmp = (HBITMAP)SelectObject(mydc, mybmp);
        BitBlt(mydc,0,0,width,height,desktopdc,0,0, SRCCOPY|CAPTUREBLT);
        SelectObject(mydc, oldbmp);
    }

    const wchar_t* filename = (argc > 1) ? argv[1] : L"screenshot.png";
    Bitmap* b = Bitmap::FromHBITMAP(mybmp, NULL);
    CLSID  encoderClsid;
    Status stat = GenericError;
    if (b && GetEncoderClsid(L"image/png", &encoderClsid) != -1) {
        stat = b->Save(filename, &encoderClsid, NULL);
    }

    if (b)
        delete b;

    // cleanup
    GdiplusShutdown(gdiplusToken);
    ReleaseDC(NULL, desktopdc);
    DeleteObject(mybmp);
    DeleteDC(mydc);
    DeleteDC(desktopdc);
    return stat == Ok;
}

Answer 1

As you mention in comments, the real problem is that the code is hanging when you try to do a screenshot of a GUI window owned by the Tcl process that is calling the screenshot program (with exec). This is because Windows wants the process to be serving its message pump — which Tcl maps to the event loop — but that message handling has stalled because Tcl is busy in a blocking wait for the subprocess to finish. (That's how exec works and how it has always worked.) This blocks everything up in a deadlock, and it sounds like the unblocking isn't handled very gracefully.

Since your program does not read any input or produce any output (well, it does do both, but by arguments and a “well-known” file) what you need is a way to run a program in the background while still being able to find out when it finishes. This means that we don't want to use exec; that only has two modes of operation: blocking wait, or totally disconnected async (if the last parameter is &). The former is the problem, and the latter doesn't give us the information we need.

Instead, we need a pipeline so that we can take the screenshot in the background while still processing events, and we need to set a fileevent so that we find out when things are finished (as that is a kind of event, under the hood).

set thepipeline [open |[list screenshot.exe $filename $windowname] r]
fileevent $thepipeline readable [list doneScreenshot $thepipeline $filename]
proc doneScreenshot {pipe filename} {
    # Pipelines become readable when either:
    #  1. there's some data to read, or
    #  2. the pipe gets closed.
    # It's option 2 that happens here.
    close $pipe
    puts "screenshot now available in $filename"
}

Since Tk-based Tcl programs run an event loop anyway, we won't use vwait to create one. (Writing your code to be asynchronous can sometimes be a bit brain-bending; think “callback” and not “program flow”. Unless you're using Tcl 8.6, when we can use a coroutine to untangle things.)

Side notes: You probably want to not pass $filename to screenshot.exe, but rather [file nativename [file normalize $filename]], but that's not the problem you're having here. And you don't need to manually add quotes around $windowname; the Tcl runtime will do that if necessary (assuming you're using the MSVC runtime for your screenshot program, rather than doing odd processing of your own).