Auto-TTL: The 2026 Frontier
Estimated read: 3 min
Predictive optimization automated the maintenance inside a table’s lifecycle – compaction, vacuuming, statistics. The next frontier platform teams are building toward is automating the end of a table’s lifecycle: deciding when data has aged past its useful life and should be demoted or deleted without a human filing a ticket. Call it auto-TTL – automatic time-to-live – and it’s less a single shipped feature than a direction several existing Databricks primitives are converging on, worth understanding now so your migration’s storage layout doesn’t have to be re-architected to take advantage of it later.
The building blocks already exist. VACUUM’s deletedFileRetentionDuration table property governs
how long deleted files stay recoverable before physical removal – that’s TTL at the file level.
Delta’s time travel window is TTL for history, not current data. What’s missing, and what
FinOps-minded platform teams are now scripting by hand with Lakeflow Jobs, is TTL at the row or
partition level: automatically archiving or dropping data past a business-defined age – 90-day-old
clickstream events, 7-year-old financial records past their regulatory retention window – driven by
policy rather than a manually scheduled DELETE. If you managed Oracle Information Lifecycle
Management (ILM) policies or Teradata’s TEMPORAL table archival, this is the same problem restated
on a lakehouse: data has a shelf life, and storage cost scales with how long you keep it beyond that.
Until a first-class policy engine ships, the pattern to build today is a scheduled Lakeflow Job that
reads a retention policy table (one row per Unity Catalog table, with a retention_days and an
action of archive or delete), and applies DELETE or a partition move against each target on a
schedule – essentially predictive optimization’s “policy at every level of the hierarchy” model,
applied to business retention rather than file compaction. Tag that policy table itself with
custom_tags so its own DBU cost is attributable, and log every run to a Delta table you can join
against system.billing.usage the same way you did for predictive optimization, so a growing
retention job cost is visible before it becomes a surprise line item.
Don’t wait for a fully automated TTL feature to start capturing the savings. A retention policy table plus a scheduled Lakeflow Job gets you 80% of auto-TTL’s benefit today, and the policy table itself becomes the interface your future automated feature would consume – build to that contract now.
The economics matter more on Databricks than they did in a fixed-capacity legacy warehouse: storage is billed continuously and separately from compute, so data with no defined retention policy is pure cost accumulation with no offsetting benefit, in a way a sunk on-prem disk array never made visible month to month.
For a migration specifically, this is where you avoid inheriting a legacy anti-pattern rather than
just automating it. Most Oracle and Teradata environments accumulated tables with no formal
retention policy at all – “keep everything, we might need it” – because on-prem storage cost was a
sunk capital expense nobody revisited quarterly. Migrating those tables as-is, with no retention
tagging, just relocates that same unbounded accumulation onto usage-based cloud storage billing,
where it becomes visible and growing on next month’s bill instead of invisible on a five-year-old
SAN. Building the retention-policy table described above as part of the migration’s schema
translation step – assigning every migrated table a retention_days value up front, even a
generous default like 730 days, rather than leaving it null – is cheaper to do once during cutover
than to retrofit across hundreds of tables a year later once someone notices the storage line item
never stopped climbing.
Reference: VACUUM and retention duration
and predictive optimization’s inheritance model
for the closest shipped analogue to policy-driven lifecycle management.
| ← Previous: Predictive Optimization: Proving Savings, Not Just Space | Next: The System-Table Query Pack and Governance Guardrails → | |:—|—:|