Airflow BigQuery Hook - Run update query via run_query

Question

I want to run some update queries after a bunch of tasks got over inside a DAG. So I'm going to create a python function for getting the necessary details from the previous task output and then by using run_query I want to update a table.

Here is my python function for this.

def metatable_update(dag,tablename,schedule,**kwargs):
    bq_hook = BigQueryHook(bigquery_conn_id='bigquery_default',location='europe-west3',use_legacy_sql=False)
    pg_conn = config[schedule][tablename][1]
    ti = kwargs['ti']
    pgmax_ts = str(ti.xcom_pull(task_ids='get_maxts_{}_{}'.format(tablename,pg_conn))[1])
    bq_hook.run_query(sql="update bqadmin.tablesync_meta set max_value='{}' where tablename='{}'".format(pgmax_ts,tablename))

My question is just simply use this code, or do I need to add something like execute or return for successfully run this task?

TheDataGuy TheDataGuy · Accepted Answer · 2021-01-05T06:18:00

Never mind, I have used BQ hook itself to solve this

And dont need to mention execute or return.

def metatable_update(dag,tablename,schedule,**kwargs):
    bq_hook = BigQueryHook(bigquery_conn_id='bigquery_default',location='europe-west3',use_legacy_sql=False)
    pg_conn = config[schedule][tablename][1]
    ti = kwargs['ti']
    pgmax_ts = str(ti.xcom_pull(task_ids='get_maxts_{}_{}'.format(tablename,pg_conn))[1])
    bq_hook.run(sql="update bqadmin.tablesync_meta set max_value='{}' where tablename='{}' and datasource_dbconn='{}'".format(pgmax_ts,tablename,pg_conn))
    return 'Executed the update query'

Airflow BigQuery Hook - Run update query via run_query

1 Answers