Compute Arbitrage: Jobs vs All-Purpose vs Serverless

Estimated read: 3 min

The single biggest lever hiding below the cost iceberg’s waterline is one most engineers assume doesn’t matter: which compute type runs a given workload. The exact same PySpark job, same cluster size, same runtime, costs meaningfully different amounts per DBU depending on whether it runs on an all-purpose cluster, jobs compute, or serverless – and because none of the three are wrong, exactly, teams routinely leave money on the table by defaulting to whichever one was easiest to click into existence first. That gap between “technically works” and “priced for what it’s actually doing” is the arbitrage.

Compute type	What it’s for	Pricing shape	Startup
All-purpose cluster	Interactive notebooks, ad hoc exploration, multiple users sharing one cluster	Highest DBU rate; billed for however long the cluster stays up	Minutes, unless already warm
Jobs compute	Scheduled, unattended pipeline runs (Lakeflow Jobs tasks)	Lower DBU rate than all-purpose for the identical instance type; cluster is created for the run and torn down after	Minutes to spin up, but only for the run’s duration
Serverless	Ad hoc SQL, notebooks, and jobs that need instant, fully managed compute with no cluster to configure	Per-second billing on Databricks-managed infrastructure; no idle cluster to forget about	Seconds

All-purpose clusters are priced for the flexibility of staying up between queries, being shared by several users, and running arbitrary interactive code – which is exactly what a data engineer needs while actively debugging a pipeline, and exactly what a scheduled 2 a.m. batch job never needs. Jobs compute exists specifically for the latter: it’s created fresh when a job starts, runs only that job’s tasks, and terminates the moment the job finishes, at a lower DBU rate than the equivalent all-purpose cluster because there’s no idle-sharing capability to pay for. Serverless compute goes further still, removing cluster configuration and startup latency almost entirely in exchange for Databricks managing the underlying infrastructure – the tradeoff being less control over instance types, which matters for a handful of specialized workloads but not for most SQL and ETL.

# Same job definition, different compute -- same output, different bill
{
  "tasks": [{
    "task_key": "nightly_load",
    "notebook_task": {"notebook_path": "/Repos/etl/load_orders"},
    "job_cluster_key": "jobs_compute"   # <- ephemeral, billed at the jobs rate
    # vs "existing_cluster_id": "0512-..."  <- an always-on all-purpose cluster
  }]
}

The arbitrage question to ask of every workload is simple: does a human need to interact with this compute session while it runs? If yes – debugging, exploring, iterating – all-purpose is the right tool and its higher rate is buying something real. If no – it’s a scheduled, unattended run – jobs compute or serverless does the identical work for less, and the only reason it isn’t already configured that way is usually habit carried over from a legacy scheduler where “the cluster” was a fixed server nobody thought to distinguish by workload type. The anti-pattern lecture later in this section walks through exactly how much that habit costs in practice.

See the compute overview for the current breakdown of serverless vs. classic (all-purpose and jobs) compute, and configuring compute for jobs for how job clusters are declared inside a Lakeflow Jobs definition.

---

| ← Previous: The Cost Iceberg: Visible Compute vs Hidden Everything | Next: Cluster Policies That Enforce Jobs Compute → | |:—|—:|