How to fetch Azure Databricks notebook run details

1
votes

I am using Azure Data Factory to run my databricks notebook, which creates job cluster at runtime, Now I want to know the status of those jobs, I mean whether they are Succeeded or Failed. So may I know, how can I get that status of runs by using job id or run id.

Note: I have not created any jobs in my databricks workspace, I am running my notebooks using Azure Data Factory which created job cluster at the runtime and it runs that notebook on top of that cluster and then it terminated that cluster

3
apache-sparkpysparkazure-data-factorydatabricksazure-databricks

3 Answers

1
votes
import json
import requests

gethooks= "https://" + databricks_instance_name + "/api/2.0/jobs/runs/list"     #add your databricks workspace instance name over here
headers={"Authorization": "Bearer ********************"}        # Add your databricks access token
response = requests.get(gethooks, headers=headers)

print(response.json())      # you will get all cluster and job related info over here in json format

# traversing through response.json
for element in response.json()['runs']:
    job_id = element['job_id']
    status = element['state']['result_state']
    job_path = element['task']['notebook_task']['notebook_path']
    job_name = job_path.split('/')
0
votes

You'll have to go to the monitor page in Azure Data Factory. You'll be able to filter by runId here.

https://docs.microsoft.com/en-us/azure/data-factory/transform-data-using-databricks-notebook#monitor-the-pipeline-run