The End-to-End Migration Runbook
Estimated read: 3 min
Every section up to this point in the course has produced a deliverable: a workload inventory, translated DDL, converted PL/SQL, a reconciliation engine, a parallel-run report, a Unity Catalog privilege map, an ABAC tag taxonomy, a FinOps query pack. A runbook is the document that stitches those deliverables into a single ordered sequence of steps – with owners, entry and exit criteria, and rollback instructions attached to each one – so that on migration day nobody is improvising from memory or reconstructing a plan from a dozen separate design docs.
Think of it the way a DBA already thinks of a major version upgrade or a data center failover runbook for an Oracle RAC cluster: a checklist that assumes the person executing it might not be the person who wrote it, might be doing it at 2 a.m., and needs every step to be unambiguous enough to follow under pressure. A migration runbook for a legacy-to-Databricks program does the same job at a much larger scale, because “the upgrade” here is actually dozens of workstreams – per-table conversions, per-domain reconciliation, per-department sign-off – that all have to land in the right order.
A runbook’s structure typically has four layers:
- Phases, the coarse-grained stages a migration actually goes through: assessment and workload inventory, schema and DDL translation, code conversion (PL/SQL to PySpark or SQL), historical backfill, parallel run, cutover, and decommission.
- Steps within each phase, each with an explicit owner, an entry criterion (what has to be true before this step can start), and an exit criterion (what has to be verified before the next step is allowed to begin).
- Rollback instructions attached to every step that touches production data or routes live traffic – not a single rollback plan at the end, but a rollback path for each irreversible-looking action.
- Escalation paths: who gets paged, and in what order, if a step’s exit criterion can’t be met.
A runbook step without an explicit, verifiable exit criterion isn’t a step – it’s a hope. “Reconciliation passed” is not an exit criterion; “the hash-diff reconciliation query returned zero variance rows across all 340 tables in the finance domain, and that result is logged” is.
In practice, the runbook is not just a Markdown or Confluence document – it’s operationalized as a Lakeflow Jobs DAG wherever a step is a Databricks-native action (a backfill job, a reconciliation query, an OPTIMIZE pass), so the same dependency structure the humans are reading is also what’s actually executing and retrying on failure. See the Lakeflow Jobs documentation for how tasks, dependencies, and retries compose into a DAG, and pair it with Databricks Asset Bundles so the jobs, schemas, and permissions the runbook depends on are deployed the same way in every environment the runbook gets rehearsed in.
The next lecture takes this runbook and turns its phase list into an actual dependency graph – what has to happen strictly in order, and what can run in parallel to compress the calendar time the whole thing takes.
| ← Previous: The Migration Playbook: Sequencing the Whole Thing | Next: Sequencing: What Order, What Parallel, What Blocks What → | |:—|—:|