Databricks Multi-Task Jobs: Orchestrating a Pipeline Without Leaving the Platform

Shannon Lowder

Shannon Lowder

— 1 min read

For two years I've been running Databricks pipelines as separate notebooks orchestrated by dbutils.notebook.run() or scheduled individually via the Jobs UI. It works, but it shows in the monitoring — five separate jobs in the UI, no unified view of the pipeline, no easy way to see that step 3 failed because step 2 returned bad data.

Multi-task Jobs changed that. Instead of five separate jobs, you define one job with five tasks and the dependencies between them. The monitoring view shows the full pipeline as a DAG, with per-task runtime and failure details. This is the version of Databricks Jobs that actually serves production pipeline needs.

Defining a Multi-Task Job

{
  "name": "daily_order_pipeline",
  "tasks": [
    {
      "task_key": "extract_raw",
      "description": "Extract from SQL Server to bronze",
      "notebook_task": {
        "notebook_path": "/pipelines/01_extract_orders",
        "base_parameters": {"processing_date": "{{ds}}"}
      },
      "new_cluster": {
        "spark_version": "7.3.x-scala2.12",
        "node_type_id": "Standard_DS3_v2",
        "num_workers": 2
      }
    },
    {
      "task_key": "transform_silver",
      "description": "Clean and conform to silver layer",
      "depends_on": [{"task_key": "extract_raw"}],
      "notebook_task": {
        "notebook_path": "/pipelines/02_transform_orders"
      },
      "new_cluster": {
        "spark_version": "7.3.x-scala2.12",
        "node_type_id": "Standard_DS3_v2",
        "num_workers": 4
      }
    },
    {
      "task_key": "aggregate_gold",
      "description": "Build daily summary metrics",
      "depends_on": [{"task_key": "transform_silver"}],
      "notebook_task": {
        "notebook_path": "/pipelines/03_aggregate_orders"
      },
      "new_cluster": {
        "spark_version": "7.3.x-scala2.12",
        "node_type_id": "Standard_DS3_v2",
        "num_workers": 2
      }
    }
  ]
}

Parallel Tasks

Tasks with the same depends_on ancestor run in parallel:

{
  "tasks": [
    {"task_key": "extract_raw", ...},
    {
      "task_key": "process_west",
      "depends_on": [{"task_key": "extract_raw"}],
      ...
    },
    {
      "task_key": "process_east",
      "depends_on": [{"task_key": "extract_raw"}],
      ...
    },
    {
      "task_key": "combine_regions",
      "depends_on": [
        {"task_key": "process_west"},
        {"task_key": "process_east"}
      ],
      ...
    }
  ]
}

process_west and process_east run simultaneously after extract_raw completes. combine_regions waits for both. This replaces the ThreadPoolExecutor pattern I was using with dbutils.notebook.run() — same outcome, but now it's visible in the Jobs UI and each task's runtime and logs are isolated.

Task-Level Retry and Timeout

{
  "task_key": "transform_silver",
  "max_retries": 2,
  "min_retry_interval_millis": 60000,  # 1 minute between retries
  "retry_on_timeout": false,
  "timeout_seconds": 3600,
  ...
}

Per-task retry configuration means a transient failure in task 2 doesn't require rerunning the entire pipeline. It retries task 2 (and only task 2) up to the configured limit. The upstream extract is preserved; the downstream aggregation waits. That's the model you want for a production pipeline. As always, I'm here to help.

Read more