Oozie: why error log shows in mapreduce job, not in Spark job?

Question

I submitted a oozie workflow that is a shell action, it calls spark-submit to run a Spring boot application which is a jar file. It runs on yarn in client mode.

However, I found that the all Spring log is inside oozie mapreduce job in yarn, not in Spark job itself. I don't understand why?

Answer 1

The oozie shell action is nothing but a map only job. By default, you spark job prints all log to console (from where it is being run). Given that that spark job is being submitted from within the oozie action, the logs are collated & visible within the shell action logs.