Backfill and Rollback Pre-Testing
Estimated read: 3 min
Every stage of the soft cutover has been described as reversible, but “reversible” is only true if the rollback path has actually been exercised before it’s needed for real. A rollback plan nobody has rehearsed is a hypothesis, not a safety net – and the worst time to discover a gap in it is during an actual incident, with the business already impacted and everyone improvising.
Rehearsing in clones is what makes practicing a rollback safe without touching production. Delta Lake’s SHALLOW CLONE creates a zero-copy snapshot of a table – metadata only, referencing the source’s existing data files – cheap enough to spin up on demand for a rollback drill. DEEP CLONE additionally copies the underlying data files, giving a fully independent copy when the drill needs to mutate data without any risk of touching the real table. See the Delta Lake clone documentation for the exact syntax and tradeoffs between the two.
-- Rehearse a rollback against an isolated copy, not production
CREATE TABLE finance.rollback_drill.ar_transactions
SHALLOW CLONE finance.gold.ar_transactions;
-- Run the rollback procedure against the clone, then verify
-- the reconciliation engine reports zero variance against
-- the pre-rollback state.
Running the actual rollback procedure against a clone, on a schedule, is what turns “we have a rollback plan” from a paragraph in a runbook into a rehearsed operational capability the team has actually done before.
Quantifying the cutover gap is the second piece. Once Databricks is the system of record (stage 2 of the soft cutover), every write it accepts before a rollback decision is made is data that doesn’t yet exist in Oracle. The cutover gap is that window – and it has to be measured, not assumed, because it’s exactly what a rollback has to account for. A gap of a few minutes with low transaction volume is a very different rollback problem than a gap of six hours during a high-volume close.
The backfill itself is the mechanism that closes that gap: replaying every transaction Databricks accepted during the gap window back into Oracle, so Oracle is caught up to the same state before it resumes as system of record. This has to be idempotent – safe to run more than once without duplicating rows – because a rollback decision made under pressure is exactly the situation where a script gets re-run by someone unsure if the first attempt succeeded. MERGE INTO is the natural tool for this: matching on the transaction’s natural key, updating what already exists and inserting only what’s genuinely new, rather than a blind INSERT that would double-count on a second run. See the MERGE INTO documentation for the full syntax.
MERGE INTO oracle_mirror.ar_transactions AS target
USING databricks_gap_window.ar_transactions AS source
ON target.transaction_id = source.transaction_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;
The backfill’s success criterion is the same reconciliation engine used everywhere else in this migration – run the hash-diff check between Oracle and Databricks immediately after a backfill and require zero variance before declaring the rollback complete. A rollback that “looks done” but hasn’t been reconciled is just an unverified claim wearing a different hat.
With a rehearsed rollback, a quantified gap, and a tested idempotent backfill in place, a cutover stage advance stops being a one-way bet – it’s a decision the team can walk back with the same confidence they walked into it. That completes the parallel-run-and-cutover toolkit; the quiz that follows checks understanding across the whole section before the course moves on to Unity Catalog and the privilege migration.
| ← Previous: Business-User Sign-Off and the Trust Gate | Next: Check Your Knowledge → | |:—|—:|