In Airflow, we use xcoms to signal that one DAG has finished and another can start running. E.g. DAG 1 runs at 12 AM and sends an xcom when it's done. DAG 2 reads the xcom from DAG 1.
One limitation of using this is that per the documentation -- xcom's can only read from DAGs with an earlier execution date, but cannot read from DAGs with a future execution date. https://airflow.apache.org/docs/stable/_modules/airflow/models/xcom.html
For the sake of an example, let's say I have 4 DAGS, DAG #1, #2, #3, #4 that all do something. Then I have DAG #5 which can run several jobs in parallel. DAG#5 relies on DAGS #1-4 for different reason and has separate checks for each of them. The DAGS are scheduled as follows:
DAG #1: 12 AM DAG #2: 12 AM DAG #3: 12 AM DAG #4: 2 AM DAG #5: 1 AM
DAG #5 has 4 tasks that are essentially utilizing the airflow.models.xcom.get_one method to read the XCcom from DAGS#1-4. But since DAG #4 has a start time of 2 AM, DAG #5 cannot read from DAG #4.
Is there a way to get around this? The only thing considered so far is simply scheduling DAG #4 earlier and having it sleep for an hour, but that doesn't seem like a good practice.
