Passing exceptions in Airflow

Question

I am new to Apache Airflow, and I plan to run Python and R script files using the BashOperator class. I want to understand how Exceptions should work in two situations:
1. The R or Python script fails for some reason; or
2. The R or Python script completes but I want to require human input before the DAG proceeds to the next task.

I have two very basic questions:
1. How does an Exception get passed from an R or Python script file to the BashOperator to the DAG? For example, should the call to an R script file be inside a try block in the BashOperator?
2. How do I pass a custom exception (warning? error?) so that even if the R or Python script completes successfully, I can pause execution of the DAG?

I'd appreciate any examples of Airflow exception handling that you could point me to.

Answer 1

Forget that you are using any Airflow operators.

Just say you are writing a Bash Script that runs the R or Python script.

Case 1: Bash Script that fails if Python Script fails:

set -e
python test_file.py

Case 2: Bash Script that passes even if Python Script fails:

python test_file.py || exit 0

Now just pass one of the above bash scripts in BashOperator.

Basically, BashOperator just runs normal Bash commands/script and passes the logs and exit status of the Script. In Case 1 your Airflow task would also fail and in case 2 the Airflow task would fail.