The Rollback Panic

Estimated read: 3 min

Run this capstone’s clock forward a second time, with one variable changed: the T+24 checkpoint fails again, but this time the row-by-row drill doesn’t find a rounding fingerprint. It finds nine rows genuinely missing from Databricks – a batch of returns processed during a brief network partition between the ingestion job and the source system, silently dropped rather than retried. That’s not a formatting quirk. That’s data loss on the table that reconciles to the $50M line, and it’s the moment this lecture is actually about: not the technical fix, but what happens in the room in the ninety seconds after someone says “we’re missing rows.”

Rollback panic is what happens when a real variance meets a team that hasn’t internalized that the rollback decision is supposed to be as procedural as the go decision was. It shows up as two opposite failure modes, and both are worse than a calm, rehearsed rollback:

Freezing. Nobody wants to be the person who calls for a rollback on a system the business just spent a year migrating to, so the group keeps investigating past the point where the checkpoint window can absorb more delay – hoping the next query will explain it away instead of escalating on schedule.
Overreacting. Someone calls for an immediate rollback before the drill finishes, on the theory that any variance on the revenue table is an automatic no-go – which would have triggered a full rollback for what, in the previous lecture, turned out to be a one-line rounding fix.

Both failure modes have the same root cause: treating the rollback decision as an emotional judgment made in the moment, instead of a procedure that was already rehearsed and already has a threshold. The backfill-and-rollback pre-testing work from the parallel-run section exists precisely to remove that judgment call from the moment of highest pressure – by the time nine rows are missing, the team should already know exactly what a rollback costs to execute (a rehearsed idempotent MERGE INTO, verified in minutes against a deep clone) and exactly what threshold triggers it, because both were decided and tested before anyone was watching a counter tick up live.

For nine missing rows discovered inside the checkpoint window, with a rehearsed rollback available and the cutover gap still small, the correct call is neither freeze nor overreact – it’s to execute the rehearsed backfill against the gap, exactly as drilled:

MERGE INTO databricks_gold.invoice_lines AS target
USING oracle_mirror.invoice_lines_gap AS source
ON target.invoice_line_id = source.invoice_line_id
WHEN NOT MATCHED THEN INSERT *;

Followed immediately by the same hash-diff check used every other time in this course: zero open mismatches, or the checkpoint doesn’t hold and the team escalates to the full rollback procedure on schedule, not on nerve.

A rollback executed on a rehearsed, tested procedure is a controlled operation with a known cost. A rollback triggered by panic – or, just as damaging, a rollback avoided by panic – is a bet made with no data behind it. The entire point of rehearsing the rollback in the parallel-run section was to make sure this decision never has to be made for the first time during an actual incident.

The steering committee still makes the final call, but they’re choosing between two well-understood options – backfill-and-continue, or invoke the full rollback – rather than reacting to a surprise. That’s the difference this whole part of the course has been building toward: not a migration with no incidents, but one where every incident has a rehearsed response instead of a live improvisation.

| ← Previous: The 48-Hour War Room: Live, No-Go, or Rollback | Next: Your Capstone Deliverable and Course Wrap → | |:—|—:|