When we run multiple flink jobs in one yarn session. We found that the logs of all jobs are written into the same file, "taskmanager.log", It is difficult for us to check logs of a specific job. Is there any approach to separate them?
Besides this, if our flink jobs are running for a long period, how to separate log files according to date?
As far as I know, there isn't anyway to separate the logs for one job, other than to run a separate cluster per job. Moreover, a lot of what is being logged isn't really job-specific.
To set up log rotation, you could put something like this in the log4j.properties file in the flink/conf directory:
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=${log.file} log4j.appender.file.MaxFileSize=1000MB
log4j.appender.file.MaxBackupIndex=0 log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
