python - stdout redirection with ctypes

Question

I'm trying to redirect the output of printf functions to a file on Windows. I'm using ctypes with python3 to invoke the functions. My code is:

import os, sys
from ctypes import *

if __name__ == '__main__':

 print("begin")
 saved_stdout=os.dup(1)
 test_file=open("TEST.TXT", "w")
 os.dup2(test_file.fileno(), 1)
 test_file.close()
 print("python print")
 cdll.msvcrt.printf(b"Printf function 1\n")
 cdll.msvcrt.printf(b"Printf function 2\n")
 cdll.msvcrt.printf(b"Printf function 3\n")
 os.dup2(saved_stdout, 1)
 print("end")

But when I run the code from Eclipse I get the following on the screen:

begin
end
Printf function 1
Printf function 2
Printf function 3

...and the following in the TEST.txt

python print

When I run this from cmd, this is what is on the screen:

begin
end

..and this is in the TEST.txt:

python print

When I comment out the second dup2() statement e.g.

import os, sys
from ctypes import *
if __name__ == '__main__':

    print("begin")
    saved_stdout=os.dup(1)
    test_file=open("TEST.TXT", "w")
    os.dup2(test_file.fileno(), 1)
    test_file.close()
    print("python print")
    cdll.msvcrt.printf(b"Printf function 1\n")
    cdll.msvcrt.printf(b"Printf function 2\n")
    cdll.msvcrt.printf(b"Printf function 3\n")
    #os.dup2(saved_stdout, 1)
    print("end")

From Eclipse, on the screen:

begin

...and in the TEST.txt file:

python print
end
Printf function 1
Printf function 2
Printf function 3

From cmd, on the screen:

begin

...and in the TEST.txt file:

python print
end

I'm totally confused now. I read all the redirection threads here on StackOverflow and I can't understand what's going on. Anyway, what I've gathered is that C functions access the stdout that is bind directly to the file descriptor, while python uses a special object for that - stdout File Object. So the elementary sys.stdout=*something* doesn't work with ctypes. I've even tried os.fdopen(1) on the dup2-ed output and then calling flush() after every printf statement but this isn't working again. I'm totally out of ideas now and would appreciate if someone have a solution for this.

related: How do I prevent a C shared library to print on stdout in python? (use your file instead of os.devnull) — jfs
I've read this thread and it's not helpful. My code is almost the same as that of the examples. I need ctypes because I'll use it later for unit testing a C library that have printfs in it. — skrech
I think that the problem will be solved if there is a way to flush the FILE pointer of the stdout, but I don't know how to access it from Python. — skrech
This is... a bizarre way of unit testing to say the very least. Or rather: your library sounds not very testable if it depends on global state and doesn't let you specify an output explicitly. It seems like the effort would be better spend on refactoring the library instead. — millimoose

Eryk Sun Eryk Sun · Accepted Answer · 2013-07-30T17:46:13

Use the same C runtime that CPython 3.x uses (e.g. msvcr100.dll for 3.3). Also include a call to fflush(NULL) before and after redirecting stdout. For good measure, redirect the Windows StandardOutput handle, in case a program uses the Windows API directly.

This can get complicated if the DLL uses a different C runtime, which has its own set of POSIX file descriptors. That said, it should be OK if it gets loaded after you've redirected Windows StandardOutput.

Edit:

I've modified the example to run in Python 3.5+. VC++ 14's new "Universal CRT" makes it much more difficult to use C standard I/O via ctypes.

import os
import sys
import ctypes, ctypes.util

kernel32 = ctypes.WinDLL('kernel32')

STD_OUTPUT_HANDLE = -11

if sys.version_info < (3, 5):
    libc = ctypes.CDLL(ctypes.util.find_library('c'))
else:
    if hasattr(sys, 'gettotalrefcount'): # debug build
        libc = ctypes.CDLL('ucrtbased')
    else:
        libc = ctypes.CDLL('api-ms-win-crt-stdio-l1-1-0')

    # VC 14.0 doesn't implement printf dynamically, just
    # __stdio_common_vfprintf. This take a va_array arglist,
    # which I won't implement, so I escape format specificiers.

    class _FILE(ctypes.Structure):
        """opaque C FILE type"""

    libc.__acrt_iob_func.restype = ctypes.POINTER(_FILE)    

    def _vprintf(format, arglist_ignored):
        options = ctypes.c_longlong(0) # no legacy behavior
        stdout = libc.__acrt_iob_func(1)
        format = format.replace(b'%%', b'\0')
        format = format.replace(b'%', b'%%')
        format = format.replace(b'\0', b'%%')
        arglist = locale = None        
        return libc.__stdio_common_vfprintf(
            options, stdout, format, locale, arglist)

    def _printf(format, *args):
        return _vprintf(format, args)

    libc.vprintf = _vprintf
    libc.printf = _printf

def do_print(label):
    print("%s: python print" % label)
    s = ("%s: libc _write\n" % label).encode('ascii')
    libc._write(1, s, len(s))
    s = ("%s: libc printf\n" % label).encode('ascii')
    libc.printf(s)
    libc.fflush(None) # flush all C streams

if __name__ == '__main__':
    # save POSIX stdout and Windows StandardOutput
    fd_stdout = os.dup(1)
    hStandardOutput = kernel32.GetStdHandle(STD_OUTPUT_HANDLE)

    do_print("begin")

    # redirect POSIX and Windows
    with open("TEST.TXT", "w") as test:
        os.dup2(test.fileno(), 1)
        kernel32.SetStdHandle(STD_OUTPUT_HANDLE, libc._get_osfhandle(1))

    do_print("redirected")

    # restore POSIX and Windows
    os.dup2(fd_stdout, 1)
    kernel32.SetStdHandle(STD_OUTPUT_HANDLE, hStandardOutput)

    do_print("end")

python - stdout redirection with ctypes

1 Answers