Lakeflow Jobs - The Orchestration Pillar
The third and final Lakeflow pillar, and Part 1’s closing section: scheduling and orchestrating everything built in Sections 8 and 9. Tasks and DAGs, building a first scheduled job, parameter passing to both notebook and Python script tasks, task values and repair runs, multi-task DAGs with conditional branching and safe backfill, automating job management via REST API/CLI and asset bundles, and the production patterns that make a job something a team actually trusts.
flowchart LR
S[Schedule / Trigger] --> J[Job]
J --> T1[Pipeline Task]
T1 --> T2[DQ Check Task]
T2 -->|pass| T3[Report Task]
T2 -->|fail| T4[Alert Task]
Lectures
| # | Lecture | Est. Read |
|---|---|---|
| 1 | What is Lakeflow Jobs - and where it fits | 8 min read |
| 2 | Your first job - pipeline task, schedule, and job-level config | 17 min read |
| 3 | Parameterizing jobs - Passing Parameters to Notebook Task | 16 min read |
| 4 | Passing Parameters to Python Script Task, Task Value, Repair Runs | 13 min read |
| 5 | Multi-task DAGs - Control Flow, Retries, and Backfill | 21 min read |
| 6 | Automating Jobs - REST API and Databricks CLI | 14 min read |
| 7 | Jobs Design Decisions and Production Patterns | 5 min read |
| 8 | Check Your Knowledge | Quiz |
| ← Previous: Check Your Knowledge | Next: What is Lakeflow Jobs - and where it fits → | |:—|—:|
Table of contents
- What is Lakeflow Jobs - and where it fits
- Your first job - pipeline task, schedule, and job-level config
- Parameterizing jobs - Passing Parameters to Notebook Task
- Passing Parameters to Python Script Task, Task Value, Repair Runs
- Multi-task DAGs - Control Flow, Retries, and Backfill
- Automating Jobs - REST API and Databricks CLI
- Jobs Design Decisions and Production Patterns
- Check Your Knowledge