How to get the Job ID and Run ID for a Databricks Job

February 23, 2023

How to get the Job ID and Run ID for a Databricks Job with working code

Sometimes there is a need to store or print system-generated values like job_id, run_id, start_time, etc. These entities are called ‘task parameter variables’. A list of supported parameters is listed here.

This is a simple 2-step process:

  1. Pass the parameter when defining the job/task

  2. Get/Fetch and print the values

Step 1: Pass the parameters

Step 2: Get/Fetch and print the values

  job_id: {dbutils.widgets.get('job_id')}
  run_id: {dbutils.widgets.get('run_id')}
  parent_run_id: {dbutils.widgets.get('parent_run_id')}
  task_key: {dbutils.widgets.get('task_key')}

Next step, when you run the job; you should see an output like this

Advanced & Optional:

In case you do not have a specific attribute in mind but want to capture the whole context information instead.

The below is code based and attributes are subject to change without notice



If you’re interested in learning more and keeping up to date with the latest about Spark, Delta, DBT, Python, SQL, Terraform, and other big data technologies, check out my other blogs and follow.