Predictive Optimization: Proving Savings, Not Just Space

Estimated read: 3 min

Predictive optimization is Databricks’ automated maintenance layer for Unity Catalog managed tables: it runs OPTIMIZE, VACUUM, and ANALYZE for you on a schedule the platform decides, instead of you hand-scheduling those jobs the way you’d have hand-scheduled an Oracle GATHER_STATS job or a Teradata COLLECT STATISTICS run. It’s been enabled by default for new accounts since November 2024, and it’s configurable by inheritance at the account, catalog, schema, or table level, which matters for a migration where some legacy-derived tables need a different retention posture than tables built natively on Databricks.

The FinOps trap with predictive optimization – and with any automated maintenance feature – is reporting its output in the wrong unit. “We reclaimed 40 TB this quarter” is a storage metric, not a savings metric, and a VP of Finance asking “so what did that save us” deserves a dollar figure, not a byte count. Proving savings requires connecting predictive optimization’s activity back to the billing tables from earlier in this section, through the dedicated system table Databricks provides for exactly this purpose:

SELECT
  h.table_name,
  h.operation_type,
  h.start_time,
  h.end_time,
  SUM(u.usage_quantity * p.pricing.default) AS optimize_run_cost_usd
FROM system.storage.predictive_optimization_operations_history h
JOIN system.billing.usage u
  ON u.usage_metadata.job_id = h.job_id
JOIN system.billing.list_prices p
  ON u.sku_name = p.sku_name AND u.cloud = p.cloud
  AND u.usage_start_time >= p.price_start_time
  AND (u.usage_start_time < p.price_end_time OR p.price_end_time IS NULL)
WHERE h.start_time >= current_date() - INTERVAL 30 DAYS
GROUP BY 1, 2, 3, 4

That query gives you the cost of running predictive optimization – the DBUs spent doing the OPTIMIZE/VACUUM/ANALYZE work itself. The savings side of the ledger is the delta it should be compared against: storage cost avoided by VACUUM reclaiming stale files, and – usually the larger number – the reduction in query-side DBU spend from better-compacted files and fresher statistics, which you measure by comparing SQL warehouse or job DBU consumption on affected tables before and after predictive optimization was enabled. That before/after comparison is the same discipline you applied in The Reconciliation Stack to prove correctness during migration, applied here to prove a dollar claim instead of a data-parity claim.

Never present predictive optimization’s maintenance cost and its downstream query savings as the same number. Report both sides – what it cost to run, and what it saved on the workloads that read the optimized tables – so the net figure survives a Finance audit.

This matters more during a migration than in a mature deployment, because newly landed tables – freshly bulk-loaded from Oracle or Teradata, with no prior compaction history – are exactly where predictive optimization has the most work to do and the most savings to prove. A table migrated as thousands of small files from a legacy export job, then compacted by predictive optimization within its first week on Databricks, is a clean before/after story: query latency on that table before compaction, query latency after, multiplied by query volume, converted to DBUs, converted to dollars. Capturing that comparison while the table is still fresh – rather than months later once nobody remembers what the pre-migration file layout looked like – is the easiest savings case you’ll ever build in this section, and it’s worth automating into the same retrospective you already run to close out each migrated table’s cutover.

Reference: Predictive optimization for Unity Catalog managed tables.

---

| ← Previous: Building the Chargeback Dashboard | Next: Auto-TTL: The 2026 Frontier → | |:—|—:|