c win32 api - GetStdHandle(STD_OUTPUT_HANDLE) is invalid, very perplexing

2
votes

I have the following fragment in my WinMain and I am launching this GUI app from the console. I want to redirect output to the console from which my app was launched.I am getting the "The handle is invalid." error after GetStdHandle().

However, if I use AllocConsole instead of AttachConsole, it works fine. In addition, if I use STD_ERROR_HANDLE instead of STD_OUTPUTHANDLE then fprintf(stderr, "errror") works fine.

I saw a blog entry which had the same problem but no solution. I am using vc 2010 compiler on 64 bit windows 7.

Thanks!

bConsole = AttachConsole(ATTACH_PARENT_PROCESS) != FALSE;

if (bConsole)
{
    int fd = 0;
    long lStdOut;
    lStdOut = (long)GetStdHandle(STD_OUTPUT_HANDLE);
    fd = _open_osfhandle(lStdOut, _O_TEXT);
    if (fd > 0)
    {
        *stdout = *_fdopen(fd, "w");
        setvbuf(stdout, NULL, _IONBF, 0 );
    }
}
printf("Test!!!!!!!!!!!!");
5
cwinapi
You need to check for errors after calling GetStdHandle. If it fails it will return INVALID_HANDLE_VALUE. Check for this and the call GetLastError if necessary. Then tell us what GetLastError reports. – David Heffernan
You should define lStdOut as intptr_t lStdOut; instead of using long lStdOut;. See msdn.microsoft.com/en-us/library/bdts1c9x.aspx – Oleg
@Oleg Not true, it's a HANDLE – see GetStdHandle – David Heffernan
@David Heffernan: Of course GetStdHandle returns a HANDLE, but _open_osfhandle use intptr_t as input parameter. One have to make type casting, but in the code of the question one uses long instead of intptr_t, but the size of long can be different from the size of HANDLE or intptr_t (having the same size as HANDLE). – Oleg
possible duplicate of Where do writes to stdout go when launched from a cygwin shell, no redirection – Ben Voigt

5 Answers

6
votes

AttachConsole does associate your process with a console, but stdout has already been opened (and connected to the old handle, whatever it was).

Overwriting stdout directly is a terrible idea. Instead, you must freopen("CONOUT$", "w", stdout); to get stdout going to the console.

But there are a lot of other little details. Have a look at my question Where do writes to stdout go when launched from a cygwin shell, no redirection which covers your problem in the question, then asks a question about some corner cases. Finally there's a code sample which incorporates everything.

5
votes

A Windows Subsystem process (i.e. one with WinMain) will not have a STDOUT, STDERR or STDIN, unless it was specifically given one on launch. The assumption is that since it is a windows program you are interacting with it via Windows.

I.e. GetStdHandle is not returning a handle to STDOUT because you haven't got a STDOUT.

To give it one, launch it thus:

winprog.exe > output.txt 2>&1

If launched in that way, it will have both a STDOUT and STDERR, which will both go into the named file.

As pointed out by other users already, AttachConsole will give you a console (nearest unix/linux equivalent is a TTY) but it will not give you a STDOUT. If you want one you will have to set it as a separate step. If you want it to be the console, you can have that too.

On the other hand a Console subsystem program (one with main) will by default have a STDIN, STDOUT and STDERR all set to the Console. You can detach the process from the console, and close them if you want.

2
votes

I added the following code into the default Visual Studio C++ GUI project, right at the start of WinMain.

if (AttachConsole(ATTACH_PARENT_PROCESS))
{
    if (GetStdHandle(STD_OUTPUT_HANDLE) == INVALID_HANDLE_VALUE)
        MessageBox(0, L"Invalid Handle", NULL, 0);
    else
        MessageBox(0, L"Valid Handle", NULL, 0);
}

When I run the GUI program from the debugger, or from Explorer, no message box shows. In other words we can't attach a console. When I run from cmd I see the "Valid Handle" message.

I conclude that there is in fact no problem with this basic approach, but that something that you are not showing us is causing the problem.

0
votes

I think your problem is here:

long lStdOut;
lStdOut = (long)GetStdHandle(STD_OUTPUT_HANDLE);
fd = _open_osfhandle(lStdOut, _O_TEXT);

I'm not very good with Win32 API, but I think handles have their own HANDLE types, which are essentially pointers I think, and 64-bit on Win64. long type in Win64 is though still 32-bit for some reason.

This is the declaration from MSDN:

HANDLE WINAPI GetStdHandle(
  __in  DWORD nStdHandle
);

Declaration for _open_osfhandle:

int _open_osfhandle (
   intptr_t osfhandle,
   int flags 
);
0
votes

To redirect output to Console use the following code:

AllocConsole();
hConsole = GetStdHandle(STD_OUTPUT_HANDLE);

then you can write to the Console using WriteFile(). 

    WriteFile(
        hConsole,                
        L"this is a debug line\n", 
        21, // string length 
        NULL,             // bytes written 
        NULL);