The Cost Iceberg and Compute Arbitrage
A successful cutover proves the platform works; it says nothing about whether it’s priced correctly. Databricks’ usage-metered billing rewards workload-aware decisions – which compute type a pipeline runs on, whether a query’s memory footprint fits Photon’s fast path, whether a cluster is even the right unit of compute for the job at hand – in a way a flat legacy license never asked anyone to think about, and that gap is exactly where a migration’s TCO promise is won or quietly lost. This section works through the specific, structural levers – compute arbitrage, enforced cluster policies, and diagnosing Photon’s disk-spill failure mode – that turn cost control from a discipline problem into a platform default, closing with the most common and most expensive habit migrated teams carry over from legacy ETL tooling.
flowchart TD
W[Workload] --> Q{"Does a human need\nto interact with it\nwhile it runs?"}
Q -->|Yes: debugging, exploring| AP[All-Purpose Cluster<br/>higher DBU rate]
Q -->|No: scheduled, unattended| JC[Jobs Compute<br/>lower DBU rate]
Q -->|No: instant, ad hoc SQL| SL[Serverless<br/>per-second billing]
AP -.enforced by.-> CP[Cluster Policy<br/>cluster_type: fixed=job]
JC -.enforced by.-> CP
JC --> PH{Photon: fits in memory?}
PH -->|Yes| Fast[Fast, low-cost run]
PH -->|No: spills to disk| Slow[Slower run,<br/>higher compute-time cost]
Slow -->|prune + cluster data,<br/>then size shuffle,<br/>hardware last| Fast
Lectures
| # | Lecture | Est. Read |
|---|---|---|
| 1 | The Cost Iceberg: Visible Compute vs Hidden Everything | 3 min read |
| 2 | Compute Arbitrage: Jobs vs All-Purpose vs Serverless | 3 min read |
| 3 | Cluster Policies That Enforce Jobs Compute | 3 min read |
| 4 | The Photon Spill | 3 min read |
| 5 | Anti-Pattern: ETL on All-Purpose Because It’s Easier to Debug | 3 min read |
| 6 | Check Your Knowledge | Quiz |
| ← Previous: Check Your Knowledge | Next: The Cost Iceberg: Visible Compute vs Hidden Everything → | |:—|—:|