GCP Pub/Sub - How to retrieve state from BQ scheduled query

Question

I have a Big Query scheduled query that triggers a cloud function via pub/sub.

I want the function to read the "state" value from the pub/sub message so I can see if it completed successfully.

The below will always trigger the else statement. If the if statement is removed, it will return a KeyError.

import base64

def hello_pubsub(event, context):
    data = base64.b64decode(event['data']).decode('utf-8')

    if 'state' in data:
        state = data['state']
        print("returned state: " + state)
    else:
        print ("No state attribute found")

Here is the pubsub message the function should receive:

{
"data":
{"dataSourceId": "scheduled_query", 
"destinationDatasetId": "xxxxxxxxxx", 
"emailPreferences": { }, 
"endTime": "2020-03-12T20:40:13.627285Z", 
"errorStatus": { },
"name": "xxxxxxxxxx", "notificationPubsubTopic": "projects/xxxxxxxxxx/topics/xxxxxxxxxx", 
"params": { "destination_table_name_template": "xxxxxxxxxx", "query": "xxxxxxxxxx", "write_disposition": "WRITE_TRUNCATE" }, 
"runTime": "2020-03-05T10:00:00Z", 
"scheduleTime": "2020-03-12T20:37:13.17166Z", 
"startTime": "2020-03-12T20:37:13.328479Z", 
"state": "SUCCEEDED", 
"updateTime": "2020-03-12T20:40:13.627307Z", 
"userId": "xxxxxxxxxx"
}
}

Show the event data received by your function in your question. What does "doesn't work` mean? An error or one of the messages? — John Hanley
Hi John, apologies for not being clearer. I've edited my post to include that information. — A Clockwork Orange
Is the update to your question the data actually received or what you think it should be? — John Hanley

A Clockwork Orange A Clockwork Orange · Accepted Answer · 2020-03-13T23:22:30

I have figured it out.

data = base64.b64decode(event['data']).decode('utf-8')

This returns a json formatted string, not a dictionary object. You need to convert to dict via:

data_dict = json.loads(data)

In order to be able to access it like a dictionary.

GCP Pub/Sub - How to retrieve state from BQ scheduled query

2 Answers