pull xcom in BigQueryOperator

Question

I'm trying to run a BigQueryOperator with some dynamic parameter based on a previous task using xcom ( I managed to push it using BashOperator with xcom_push=True)

I thought using the following would do the trick

def get_next_run_date(**context):
    last_date = context['task_instance'].xcom_pull(task_ids=['get_autoplay_last_run_date'])[0].rstrip()
    last_date = datetime.strptime(last_date, "%Y%m%d").date()
    return last_date + timedelta(days=1)

t3 = BigQueryOperator(
    task_id='autoplay_calc',
    bql='autoplay_calc.sql',
    params={
            "env" : deployment
            ,"region" : region
            ,"partition_start_date" : get_next_run_date()
            },
    bigquery_conn_id='gcp_conn',
    use_legacy_sql=False,
    write_disposition='WRITE_APPEND',
    allow_large_results=True,
    #provide_context=True,
    destination_dataset_table=reporting_project + '.pa_reporting_public_batch.autoplay_calc',
    dag=dag
    )` 

but using the above provide me with a Broken Dag Error with 'task_instance' error.

Answer 1

Have you tried using context['ti'].xcom_pull()?

Answer 2

You are using it in a wrong way.

You can not use xcom in params. You need to use it in bql/sql parameter. You sql file, autoplay_calc.sql can contain something like

select * from XYZ where date == "{{xcom_pull(task_ids=['get_autoplay_last_run_date'])[0].rstrip() }}"