I have a logger that has a RotatingFileHandler
.
I want to redirect all Stdout
and Stderr
to the logger.
How to do so?
8 Answers
Not enough rep to comment, but I wanted to add the version of this that worked for me in case others are in a similar situation.
class LoggerWriter:
def __init__(self, level):
# self.level is really like using log.debug(message)
# at least in my case
self.level = level
def write(self, message):
# if statement reduces the amount of newlines that are
# printed to the logger
if message != '\n':
self.level(message)
def flush(self):
# create a flush method so things can be flushed when
# the system wants to. Not sure if simply 'printing'
# sys.stderr is the correct way to do it, but it seemed
# to work properly for me.
self.level(sys.stderr)
and this would look something like:
log = logging.getLogger('foobar')
sys.stdout = LoggerWriter(log.debug)
sys.stderr = LoggerWriter(log.warning)
UPDATE for Python 3:
- Including a dummy flush function which prevents an error where the function is expected (Python 2 was fine with just
linebuf=''
). - Note that your output (and log level) appears different if it is logged from an interpreter session vs being run from a file. Running from a file produces the expected behavior (and output featured below).
- We still eliminate extra newlines which other solutions do not.
class StreamToLogger(object):
"""
Fake file-like stream object that redirects writes to a logger instance.
"""
def __init__(self, logger, level):
self.logger = logger
self.level = level
self.linebuf = ''
def write(self, buf):
for line in buf.rstrip().splitlines():
self.logger.log(self.level, line.rstrip())
def flush(self):
pass
Then test with something like:
import StreamToLogger
import sys
import logging
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
filename='out.log',
filemode='a'
)
log = logging.getLogger('foobar')
sys.stdout = StreamToLogger(log,logging.INFO)
sys.stderr = StreamToLogger(log,logging.ERROR)
print('Test to standard out')
raise Exception('Test to standard error')
See below for old Python 2.x answer and the example output:
All of the prior answers seem to have problems adding extra newlines where they aren't needed. The solution that works best for me is from http://www.electricmonk.nl/log/2011/08/14/redirect-stdout-and-stderr-to-a-logger-in-python/, where he demonstrates how send both stdout and stderr to the logger:
import logging
import sys
class StreamToLogger(object):
"""
Fake file-like stream object that redirects writes to a logger instance.
"""
def __init__(self, logger, log_level=logging.INFO):
self.logger = logger
self.log_level = log_level
self.linebuf = ''
def write(self, buf):
for line in buf.rstrip().splitlines():
self.logger.log(self.log_level, line.rstrip())
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
filename="out.log",
filemode='a'
)
stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl
stderr_logger = logging.getLogger('STDERR')
sl = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = sl
print "Test to standard out"
raise Exception('Test to standard error')
The output looks like:
2011-08-14 14:46:20,573:INFO:STDOUT:Test to standard out
2011-08-14 14:46:20,573:ERROR:STDERR:Traceback (most recent call last):
2011-08-14 14:46:20,574:ERROR:STDERR: File "redirect.py", line 33, in
2011-08-14 14:46:20,574:ERROR:STDERR:raise Exception('Test to standard error')
2011-08-14 14:46:20,574:ERROR:STDERR:Exception
2011-08-14 14:46:20,574:ERROR:STDERR::
2011-08-14 14:46:20,574:ERROR:STDERR:Test to standard error
Note that self.linebuf = ''
is where the flush is being handled, rather than implementing a flush function.
If it's an all-Python system (i.e. no C libraries writing to fds directly, as Ignacio Vazquez-Abrams asked about) then you might be able to use an approach as suggested here:
class LoggerWriter:
def __init__(self, logger, level):
self.logger = logger
self.level = level
def write(self, message):
if message != '\n':
self.logger.log(self.level, message)
and then set sys.stdout
and sys.stderr
to LoggerWriter
instances.
You can use redirect_stdout context manager:
import logging
from contextlib import redirect_stdout
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.write = lambda msg: logging.info(msg) if msg != '\n' else None
with redirect_stdout(logging):
print('Test')
or like this
import logging
from contextlib import redirect_stdout
logger = logging.getLogger('Meow')
logger.setLevel(logging.INFO)
formatter = logging.Formatter(
fmt='[{name}] {asctime} {levelname}: {message}',
datefmt='%m/%d/%Y %H:%M:%S',
style='{'
)
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(formatter)
logger.addHandler(ch)
logger.write = lambda msg: logger.info(msg) if msg != '\n' else None
with redirect_stdout(logger):
print('Test')
As an evolution to Cameron Gagnon's response, I've improved the LoggerWriter
class to:
class LoggerWriter(object):
def __init__(self, writer):
self._writer = writer
self._msg = ''
def write(self, message):
self._msg = self._msg + message
while '\n' in self._msg:
pos = self._msg.find('\n')
self._writer(self._msg[:pos])
self._msg = self._msg[pos+1:]
def flush(self):
if self._msg != '':
self._writer(self._msg)
self._msg = ''
now uncontrolled exceptions look nicer:
2018-07-31 13:20:37,482 - ERROR - Traceback (most recent call last):
2018-07-31 13:20:37,483 - ERROR - File "mf32.py", line 317, in <module>
2018-07-31 13:20:37,485 - ERROR - main()
2018-07-31 13:20:37,486 - ERROR - File "mf32.py", line 289, in main
2018-07-31 13:20:37,488 - ERROR - int('')
2018-07-31 13:20:37,489 - ERROR - ValueError: invalid literal for int() with base 10: ''
Output Redirection Done Right!
The Problem
logger.log
and the other functions (.info
/.error
/etc.) output each call as a separate line, i.e. implicitly add (formatting and) a newline to it.
sys.stderr.write
on the other hand just writes its literal input to stream, including partial lines. For example: The output "ZeroDivisionError: division by zero" is actually 4(!) separate calls to sys.stderr.write
:
sys.stderr.write('ZeroDivisionError')
sys.stderr.write(': ')
sys.stderr.write('division by zero')
sys.stderr.write('\n')
The 4 most upvoted approaches (1, 2, 3, 4) thus result in extra newlines -- simply put "1/0" into your program and you will get the following:
2021-02-17 13:10:40,814 - ERROR - ZeroDivisionError
2021-02-17 13:10:40,814 - ERROR - :
2021-02-17 13:10:40,814 - ERROR - division by zero
The Solution
Store the intermediate writes in a buffer. The reason I am using a list as buffer rather than a string is to avoid the Shlemiel the painter’s algorithm. TLDR: It is O(n) instead of potentially O(n^2)
class LoggerWriter:
def __init__(self, logfct):
self.logfct = logfct
self.buf = []
def write(self, msg):
if msg.endswith('\n'):
self.buf.append(msg.removesuffix('\n'))
self.logfct(''.join(self.buf))
self.buf = []
else:
self.buf.append(msg)
def flush(self):
pass
# To access the original stdout/stderr, use sys.__stdout__/sys.__stderr__
sys.stdout = LoggerWriter(logger.info)
sys.stderr = LoggerWriter(logger.error)
2021-02-17 13:15:22,956 - ERROR - ZeroDivisionError: division by zero
For versions below Python 3.9, you could replace replace .removesuffix('\n')
with the less accurate .rstrip('\n')
.
Quick but Fragile One-Liner
sys.stdout.write = logger.info
sys.stderr.write = logger.error
What this does is simply assign the logger functions to the stdout/stderr .write
call which means any write call will instead invoke the logger functions.
The downside of this approach is that both calls to .write
and the logger functions typically add a newline so you will end up with extra lines in your log file, which may or may not be a problem depending on your use case.
Another pitfall is that if your logger writes to stderr itself we get infinite recursion (a stack overflow error). So only output to a file.
stdout
andstderr
message to my logger. - orenma