Skip to main content

Managing Partitions

Partitions control which nodes are available to specific groups of users and set resource limits. Partitions are defined in the Slinky Controller CR's spec.extraConf field.

Target nodes by node set, not by node name. Every node set in your cluster is automatically published to Slurm as a nodeset you can reference directly in a partition's Nodes= field, and that reference keeps working as nodes are replaced, added, or removed.

Why Not Node Names​

Node names are not stable. When a node is replaced, its replacement comes back under a new name, and a partition that still lists the old name can stop the Slurm controller from starting. That affects scheduling across the whole cluster, not just the one partition.

warning

Do not list node names or ranges such as Nodes=np-2e2792bc-[1-2] in a partition. Use a node set name or Nodes=ALL, both of which stay valid as nodes come and go.

Find Your Node Set Names​

Each node set is published to Slurm under the name <slurm-cluster-name>-<node-set-name>. A node set called h200-workers on a cluster called research is available to partitions as research-h200-workers.

To list the names available on your cluster, run this from a login node:

sinfo -h -o "%f" | sort -u
research-cpu-workers
research-h200-workers

Each line is a name you can use in Nodes=. To see which nodes belong to each one, add the node list:

sinfo -h -o "%f %N" | sort -u
research-cpu-workers np-7e02223a-1
research-h200-workers np-2e2792bc-[1-2]

Creating a Partition​

Step 1 — Edit the Controller CR

The Controller CR is named slurm-<cluster-name> in the slurm namespace:

kubectl -n slurm edit controller slurm-<cluster-name>

Step 2 — Add partition lines to spec.extraConf

The spec.extraConf field contains both your custom configuration and an automatically injected section managed by the Crusoe Slurm Operator. Add your PartitionName= lines above the injected section markers:

spec:
  extraConf: |
    PartitionName=ml-team Nodes=research-h200-workers MaxTime=08:00:00 State=UP

    # THE FOLLOWING SETTINGS ARE AUTOMATICALLY INJECTED BY CRUSOE SLURM OPERATOR
    # ===============================START======================================
    SlurmctldDebug=debug5
    SlurmdDebug=debug5
    ...
    PartitionName=all Nodes=ALL Default=YES MaxTime=UNLIMITED State=UP
    # ================================END=======================================
warning

Do not modify anything between the START and END markers. The operator overwrites this section on every reconciliation cycle.

note

If you add Default=YES to your custom partition, the operator will automatically remove Default=YES from the all partition in the injected section.

Step 3 — Verify the partition

From a login node, run sinfo to confirm the new partition is available:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
all*         up   infinite      3   idle np-2e2792bc-[1-2],np-7e02223a-1
ml-team      up    8:00:00      2   idle np-2e2792bc-[1-2]

Step 4 — Reconfigure Slurm (optional)

If Slurm doesn't pick up the change automatically, run from a login node:

scontrol reconfigure

Splitting Compute Across Partitions​

A node set is the smallest unit of partition membership. You cannot split one node set between two partitions and have the split survive node replacement, so plan node sets around how you intend to partition compute. If a team or workload needs its own partition, give it its own node set.

Point each partition at the node set that backs it:

PartitionName=ml-team   Nodes=research-h200-workers   MaxTime=08:00:00 State=UP
PartitionName=cpu-tasks Nodes=research-cpu-workers    MaxTime=UNLIMITED State=UP

Several partitions can share one node set, which is the usual way to offer the same hardware at different priorities:

PartitionName=high   Nodes=research-h200-workers Default=NO  MaxTime=UNLIMITED State=UP PriorityTier=100
PartitionName=normal Nodes=research-h200-workers Default=YES MaxTime=UNLIMITED State=UP PriorityTier=10
PartitionName=low    Nodes=research-h200-workers Default=NO  MaxTime=UNLIMITED State=UP PriorityTier=1

A partition can also span several node sets. List them comma separated, and the partition contains every node from each:

PartitionName=everything Nodes=research-h200-workers,research-cpu-workers MaxTime=UNLIMITED State=UP
note

A partition that references a node set with no registered nodes is not an error. It appears in sinfo with zero nodes and starts scheduling as soon as nodes join. This is what makes node set references safe to configure before capacity is attached.

Next Steps​