How to get the Run id's using Job Id using Databricks CLI

Question

I tried to get the Run id using databricks runs list on CLI but didn't got the Run id's of all the jobs that run's everyday i got only top 20 Run id's but then i got the Job id of all jobs using databricks jobs list --output json now i want to get the Run id's of all job using Job id's. please help me with this i'm new to databricks.

Answer 1

Unfortunately, databricks cli doesn't provide the information of the Run ids.

Note: Only jobs started by the Databricks executor display using the job ID specified in the stage. The job ID is the same for all instances of the job.

You can find the run ID for a particular instance in the Data Collector log.

The Databricks executor also writes the run ID of the job to the event record. To keep a record of all run IDs, enable event generation for the stage.

There are different methods to get the RunId for any given job:

Azure Databricks Portal (user Interface): By clicking on the Jobs tab, you can view all the Jobs which you have created.

Select any Job to get detailed RunId for each run.

Azure Portal (user Interface) Using Kusto Query Language: If you have configured diagnostic log delivery, you can use KQL queries to get the JobID and RunID:

Databricks REST API: You can use the below REST API command to get list of jobs and runs.

curl "https://centralus.azuredatabricks.net/api/2.0/jobs/runs/list" -X GET -H "Authorization: Bearer dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXXX4a"