Airflow 1.9 over verbose logging

Question

After upgrading from version 1.7.1.3 I noticed more verbose logging messages in Airflow tasks. To be more precise, my current airflow 1.9 output message has following format when I am running bash bash operator task:

[2018-05-17 16:43:08,104] {base_task_runner.py:98} INFO - Subtask: [2018-05-17 16:43:08,104] {bash_operator.py:101} INFO - <SCRIPT LOGS HERE> While on 1.7.1.3 the messages had following format: [2018-05-17 16:10:02,615] {bash_operator.py:77} INFO - <SCRIPT LOGS HERE>

Is there any way to return to previous level of log details (from v. 1.7.1.3) on airflow 1.9, i.e. Not display base_task_runner logs in config?

I have tried to modify log format in airflow.cfg

# Logging class
# Specify the class that will specify the logging configuration
# This class has to be on the python classpath
# logging_config_class = my.path.default_local_settings.LOGGING_CONFIG
logging_config_class =

# Log format
log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s

namely I tried to modify remove asctime from log_format, but that was removing timestamps from both base_task_runner and bash_operator. Maybe simple_log_format could solve this? What is the difference between log_format and simple_log_format variables?

I also haven't set up logging config class. I've got an impression that was mainly used for pushing the logs remotely do I still need it if I store my logs locally?

Thanks

tobi6 tobi6 · Accepted Answer · 2018-05-18T11:23:31

I think this is not possible because some calling structures have changed between versions, if I am not mistaken.

Task calls will always be a Subtask. Since this means a different hierarchy, the log structure is also affected.

Airflow 1.9 over verbose logging

1 Answers