The Photon Spill

Estimated read: 3 min

Photon is Databricks’ native, vectorized query engine – written in C++ rather than running on the JVM, processing data in columnar batches instead of row by row – and it accelerates SQL, DataFrame operations, and ETL pipelines without requiring any code changes. It’s on by default on modern runtimes, and the natural assumption is that turning it on only ever helps. That assumption breaks the moment a shuffle, join, or aggregation needs more memory than Photon’s off-heap allocation has available: instead of failing, the operation spills to disk, and disk I/O is an order of magnitude slower than the in-memory path Photon was built for. A query that Photon should have accelerated ends up running longer than it would have without Photon at all – and because Databricks bills by compute-time, a spilling query doesn’t just run slow, it costs more for the privilege.

This is the trap in the name: Photon spill isn’t a Photon bug, it’s memory pressure that Photon happens to make more visible, because Photon’s performance ceiling is high enough that the gap between “fits in memory” and “spills to disk” shows up as a much bigger swing than it would on the row-based engine it replaced. Diagnosing it means opening a query’s query profile and checking the spill metric on the join or shuffle stage – a nonzero spill number is the signal, not the CPU or wall-clock time alone.

The fix order matters, because each step is progressively more expensive to get wrong:

Prune and cluster the data first. Most spill comes from scanning and shuffling far more data than the query actually needs. Liquid Clustering (from the physical design section earlier in this part) keeps related rows co-located on disk, so a filtered query reads less data and has less to shuffle in the first place.
```
ALTER TABLE finance.transactions.orders CLUSTER BY (order_date, region);
OPTIMIZE finance.transactions.orders;
```
Size the shuffle correctly. A join or aggregation with too few shuffle partitions concentrates too much data per task, exceeding what fits in memory even after pruning; too many partitions adds overhead without helping. spark.sql.shuffle.partitions and checking for data skew in the query profile (one partition doing far more work than its siblings) are the levers here, before touching hardware at all.
Only then, consider hardware. Scaling up to a larger instance type or adding memory-optimized nodes fixes the symptom – more headroom before spill kicks in – without touching the cause, and it’s priced accordingly. It’s a legitimate last resort for a workload that’s genuinely memory-intensive by nature, not a first response to a query that’s spilling because it’s scanning an unclustered table.

Reach for a bigger cluster only after pruning and shuffle sizing have been tried – hardware is the most expensive lever in this list, and it’s the one most engineers reach for first because it requires no query changes. That ordering is backwards from a cost-arbitrage standpoint: it treats the most expensive fix as the default rather than the last resort.

See the Photon documentation for supported operators and current limitations, and the query profile guide for reading spill and memory metrics off a specific query run.

| ← Previous: Cluster Policies That Enforce Jobs Compute | Next: Anti-Pattern: ETL on All-Purpose Because It’s Easier to Debug → | |:—|—:|