The 48-Hour War Room: Live, No-Go, or Rollback

Estimated read: 3 min

Every gate on the matrix is green. Cutover executes. Twelve hours into the 72-hour rollback window, the reconciliation dashboard’s big counter – the one built to stay at zero – ticks up: three open mismatches on order_to_cash.invoice_lines, the table that reconciles to the $50M revenue line. This is the war room the section’s title promises: not a scheduled meeting, but everyone who owns a piece of this cutover in one channel, working one problem, with the T+24 checkpoint from the rollback-window structure as the clock.

The heat map from the reconciliation dashboards localizes it fast – one table, one column, line_total. That’s the reconciliation stack doing exactly what it’s for: not just saying something’s wrong, but pointing at where. The next step is a row-by-row drill: pull the specific rows the hash-diff flagged and compare them side by side instead of trusting the aggregate.

SELECT o.invoice_line_id, o.line_total AS oracle_total, d.line_total AS databricks_total,
       o.line_total - d.line_total AS delta
FROM oracle_mirror.invoice_lines o
JOIN databricks_gold.invoice_lines d USING (invoice_line_id)
WHERE ABS(o.line_total - d.line_total) > 0;

Every flagged row shows a delta of exactly 0.01 or -0.01, and every one of them has a fractional cent that lands on exactly .5 before rounding – 12.005, 8.995, and so on. That pattern is the fingerprint of a rounding-mode mismatch, not a logic bug: the migrated procedure’s PySpark translation used Python’s built-in round() on the way from PL/SQL, and Python’s round() uses round-half-to-even (banker’s rounding) on exact ties, while Oracle’s ROUND function – and the business logic every downstream report was built to match – uses round-half-away-from-zero. 12.005 rounds to 12.00 under Python’s default and 12.01 under Oracle’s, and the two disagree on every tied case, invisibly, until a reconciliation job with cent-level precision catches it.

The fix is one line: replace the Python built-in with pyspark.sql.functions.round, which implements HALF_UP – Spark SQL’s round-half-away-from-zero, matching Oracle’s behavior – instead of relying on Python semantics that happened to run inside a PySpark job.

from pyspark.sql import functions as F
# Wrong: Python's round() ties to even -- diverges from Oracle on exact .5 cents
df = df.withColumn("line_total", F.expr("python_round(raw_total, 2)"))
# Correct: Spark SQL ROUND is HALF_UP, matching Oracle's ROUND
df = df.withColumn("line_total", F.round("raw_total", 2))

This is exactly the kind of gap the earlier section on pattern translation warned about: a transpiled procedure can run without error, produce plausible-looking numbers, and still diverge from the source system on a narrow, silent edge case that only a row-level reconciliation check – not a smoke test, not a spot check – will ever surface.

Fixing the function and re-running the affected batch closes the gap on new writes, but the rows already written during the mismatch window still need a delta sync: an idempotent MERGE INTO backfill, the same mechanism rehearsed in the parallel-run section, correcting only the rows the drill flagged rather than reprocessing the whole table. Once that backfill runs, the hash-diff check goes back to zero open mismatches, and the T+24 checkpoint holds – clean, on time, with the root cause documented rather than just patched. The steering committee’s go/no-go call at T+72 is now made on a matrix where every row, including the one that broke, is backed by a rerunnable fix – not a promise that “it’s fine now.”

---

| ← Previous: The Go or No-Go Decision Matrix | Next: The Rollback Panic → | |:—|—:|