Cluster Policies That Enforce Jobs Compute

Estimated read: 3 min

Knowing that jobs compute is cheaper than all-purpose for unattended workloads doesn’t, by itself, change anyone’s behavior. The engineer who built a pipeline last year, pointed it at an existing all-purpose cluster because it was already running, and moved on has no reason to revisit that choice – the pipeline works, and “works” is usually where attention stops. Compute arbitrage only pays off if the platform makes the cheap path the default path, not an optional best practice someone has to remember. That’s what a cluster policy is for.

A cluster policy is a JSON rule set, attached to a group or user, that constrains what compute they’re allowed to create – which fields are fixed, which are restricted to an allowlist, and which are forbidden entirely. Rather than trusting every pipeline author to choose jobs compute correctly, a policy can make all-purpose clusters simply unavailable to the service principal that runs scheduled jobs:

{
  "cluster_type": {
    "type": "fixed",
    "value": "job"
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 15,
    "hidden": true
  },
  "node_type_id": {
    "type": "allowlist",
    "values": ["i3.xlarge", "i3.2xlarge", "i3.4xlarge"]
  },
  "spark_version": {
    "type": "fixed",
    "value": "auto:latest-lts"
  }
}

The cluster_type: fixed, value: job rule is the enforcement point that matters most for this section’s argument: it makes it structurally impossible for a compute created under this policy to become a long-lived, shared all-purpose cluster, regardless of what any individual pipeline author intends. Pair that with a fixed, non-editable autotermination_minutes (the hidden: true flag hides the field from the UI entirely, rather than just defaulting it) to close the other common leak – an all-purpose cluster left running overnight because nobody set a low enough auto-termination window before walking away.

Cluster policies are enforced at cluster creation, not applied retroactively to running clusters – rolling out a jobs-compute-only policy stops new sprawl immediately but does nothing to the all-purpose clusters already running under the old rules. Pair a new policy rollout with an audit of existing always-on clusters, or the fix only affects work built after the policy date.

This is the same governance pattern as the tag-based ABAC policies from the last section, applied to cost instead of data access: instead of relying on every engineer remembering the right choice, the platform encodes the rule once and enforces it structurally. See the compute policies overview and the policy definition reference for the full set of policy types (fixed, allowlist, blocklist, range, regex) beyond the ones shown here.

---

| ← Previous: Compute Arbitrage: Jobs vs All-Purpose vs Serverless | Next: The Photon Spill → | |:—|—:|