QoS available on the ARIES cluster

Overview

On the ARIES cluster, the Quality of Service (QoS) configurations in Slurm define priorities and limits that can be applied to jobs. The currently available QoS entries are:

• normal
• high
• low
• nolimits
• gpus
• serial
• unsafe_qos

The data presented on this page is derived from the output of:

sacctmgr show qos

Important note: in the provided output, some TRES values may be abbreviated with the + character. This indicates that sacctmgr has truncated the contents of the column for display purposes. To avoid incorrect interpretations, the values are reported exactly as displayed.

QoS summary

Meaning of the QoS parameters

Slurm Quality of Service (QoS) configurations make it possible to define priorities, resource limits, maximum numbers of jobs, and preemption policies.

The main parameters used in the cluster QoS configurations are described below.

Preempt

Specifies which QoS entries may be subject to preemption by the current QoS.

Preemption allows a job belonging to a QoS with higher priority or privileges to affect jobs belonging to the QoS entries specified in this field.

If the field is empty, no QoS is configured to be preempted through this parameter.

Example:

Preempt = unsafe_qos

This indicates that the current QoS can apply its preemption policy to jobs belonging to the unsafe_qos QoS.

PreemptMode

Defines how job preemption is performed.

Some possible values are:

• CANCEL: the job is terminated.
• REQUEUE: the job is terminated and placed back into the queue, if it can be requeued.
• SUSPEND: the job is suspended.
• OFF: disables preemption for the QoS.

The actual behavior also depends on the global Slurm configuration and on the PreemptType configured on the cluster.

Flags

Contains additional options associated with the QoS.

On the ARIES cluster, one of the flags used is:

DenyOnLimit

DenyOnLimit causes a job that violates certain QoS limits to be rejected rather than simply remaining pending until the required resources or limits become available.

This makes it possible to immediately inform the user that the submitted request is not compatible with the limits of the selected QoS.

MaxTRES

Defines the maximum amount of Trackable RESources (TRES) that can be used by a single job belonging to the QoS.

TRES represent resources that Slurm can track, account for, and limit, for example:

• cpu: number of CPUs;
• mem: memory;
• node: number of nodes;
• gres/gpu: number of GPUs.

Example:

cpu=8,gres/gpu=4,mem=100G

This means that a single job subject to this QoS may be limited to:

• 8 CPUs;
• 4 GPUs;
• 100 GB of memory.

MaxWall

Defines the maximum execution time (walltime) allowed for a job belonging to the QoS.

Example:

12-12:00:00

This uses the format:

days-hours:minutes:seconds

and therefore corresponds to:

12 days and 12 hours

A job using this QoS cannot request a --time value greater than the configured limit.

MaxTRESPU

MaxTRESPU stands for:

Maximum TRES Per User

It defines the maximum total amount of TRES resources that a single user can use simultaneously across jobs belonging to the QoS.

Unlike MaxTRES, which limits the resources available to an individual job, MaxTRESPU applies the limit to the combined jobs of a user.

Example:

gres/gpu=3,node=10

This specifies per-user limits of:

• 3 GPUs;
• 10 nodes.

The limit therefore applies to the user's aggregate resource usage within the QoS and not necessarily to a single job.

MaxJobsPU

MaxJobsPU stands for:

Maximum Jobs Per User

It defines the maximum number of jobs that a single user can have running simultaneously within the QoS.

Example:

MaxJobsPU = 8

This means that each user can have at most 8 jobs running simultaneously using that QoS.

Additional submitted jobs remain pending, provided that the MaxSubmitPU limit has not also been exceeded.

MaxSubmitPU

MaxSubmitPU stands for:

Maximum Submit Jobs Per User

It defines the maximum number of jobs that a single user can have submitted to Slurm at the same time within the QoS.

The count therefore includes jobs still present in the system and subject to the QoS, not only the jobs that are currently running.

Example:

MaxJobsPU   = 8
MaxSubmitPU = 24

This means that a user can have:

• at most 8 jobs running simultaneously;
• at most 24 jobs submitted in total under the QoS.

MaxSubmitPU therefore helps prevent a single user from submitting an excessive number of jobs to the queue.

MaxJobsPA

MaxJobsPA stands for:

Maximum Jobs Per Account

It defines the maximum number of jobs that can run simultaneously per Slurm account within the QoS.

Unlike MaxJobsPU, this limit is not applied to an individual user but to the entire account.

Example:

MaxJobsPA = 104

This means that all users belonging to the same account share an overall limit of 104 simultaneously running jobs for that QoS.

MaxSubmitPA

MaxSubmitPA stands for:

Maximum Submit Jobs Per Account

It defines the maximum number of jobs that can be submitted to Slurm by an account at the same time, considering all users belonging to that account.

Example:

MaxSubmitPA = 104

This indicates that the account can have up to 104 submitted jobs in total using that QoS.

The limit is shared among all users belonging to the account.

Difference between PU and PA limits

The suffixes used by these Slurm parameters make it possible to immediately identify the scope of a limit:

For example:

MaxJobsPU = 8

limits each individual user to 8 running jobs, while:

MaxJobsPA = 104

limits the entire account to 104 running jobs, regardless of how those jobs are distributed among the users belonging to the account.

Difference between MaxTRES and MaxTRESPU

It is important to distinguish between these two parameters:

For example, if a QoS defines:

MaxTRES    = gres/gpu=4
MaxTRESPU  = gres/gpu=4

a single job cannot use more than 4 GPUs and, at the same time, all jobs belonging to the same user and subject to this QoS cannot exceed an aggregate limit of 4 GPUs.

Summary

Using a QoS in a Slurm job script

To request a QoS in a Slurm job, use the following directive:

#SBATCH --qos=<qos_name>

Example:

#!/bin/bash
#SBATCH --job-name=example
#SBATCH --qos=normal
#SBATCH --time=01:00:00
#SBATCH --nodes=1

srun ./program

The requested QoS must be allowed for the Slurm association of the user and the account used by the job.

Checking the complete values

To display the fields without abbreviations caused by column width, it is recommended to use parsable output:

sacctmgr show qos -P

To restrict the output to the main fields:

sacctmgr show qos format=Name,Priority,Preempt,PreemptMode,Flags,UsageFactor,GrpTRES,MaxTRES,MaxTRESPU,MaxJobsPU,MaxSubmitPU