To start using the cluster, you'll need to first prepare a batch script. A batch script is a file that's created using an editor, such as vim or emacs. It contains instructions that tell the scheduler the resources and steps for your job. If you are new to creating and editing files in the linux environment, or linux in general, visit: https://www.digitalocean.com/community/tutorials/an-introduction-to-linux-basics
Once you've prepared your batch script, you will submit it to the scheduler as a job.
A job is an allocation of resources assigned to an individual user for a specified amount of time. When a job runs, the scheduler selects and allocates resources to the job. The invocation of the application happens within the batch script, or at the command line for interactive and jobs. Job steps are sets of (possibly parallel) tasks within a job.
Preparing your batch script
A batch job script is composed of three main components:
- The interpreter used to execute the script
#SBATCHdirectives that convey Slurm scheduler options. (Note: a#in Slurm is not treated as a comment)- The application(s) to execute along with its input arguments and options
Let's look at an example batch script containing the previously mentioned components:
#!/bin/bash # COMMENT:The interpreter used to execute the script
# COMMENT: #SBATCH directives that convey submission options:
#SBATCH --job-name=example_job
#SBATCH [email protected]
#SBATCH --mail-type=BEGIN,END
#SBATCH --cpus-per-task=1
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --mem-per-cpu=1000m
#SBATCH --time=10:00
#SBATCH --account=test
#SBATCH --partition=standard
#SBATCH --output=/home/%u/%x-%j.log
# COMMENT:The application(s) to execute along with its input arguments and options:
<insert commands here>
Please note: if your job is set to utilize more than one node, make sure your code is MPI enabled in order to run across these nodes. You can use any of srun, mpirun, or mpiexec commands to start your MPI job.
For example, if you want to run a simple MPI job you would replace <insert commands here> with something like:
srun -N4 --mpi=pmix ./hello
The next section has more detail on these and other common job submission options one might include in their batch script.
For a more exhaustive list of options see Slurm's sbatch documentation. Also see Slurm's MPI documentation.
COMMON JOB SUBMISSION OPTIONS
| Option | Slurm Command (#SBATCH) | Example Usage |
| Job name | --job-name=<name> |
--job-name=gljob1 |
| Account | --account=<account> |
This should be set to the account that will pay for the work. In most cases this will be a UMRCP account, a researcher provider account, class account, or unit based account (LSA, COE, etc). |
| Queue | --partition=<name> |
--partition=standard
Available partitions: standard (default), gpu (GPU jobs only), largemem (large memory jobs only), viz, debug, standard-oc (on-campus software only) |
| Wall time limit | --time=<dd-hh:mm:ss> |
Typically, the MaxTime value on a partition will dictate how long your job can run. If you're unsure of how much time to ask for, you can test for up to 4 hours on the debug partition. Common pitfalls can be: not requesting enough wall time, resulting in your job terminating too soon, and requesting too much wall time which could delay the running of your job. For additional assistance, you can reach out to your unit support HPC staff. |
| Node count | --nodes=<count> |
If not specified, the default behavior is to allocate enough nodes to satisfy the requested resources as expressed by per-job specification options (e.g. --ntasks, --cpus-per-tasks). For additional assistance determining your node geometry, you can reach out to your unit support HPC staff. |
| Process count per node | --ntasks-per-node=<count> |
Request that ntasks be invoked on each node. If you're doing MPI work, this will be relevant to you. |
| Core count (per process) | --cpus-per-task=<cores> |
Without this option, Slurm will just try to allocate one processor per task. |
| Memory limit | --mem=<limit> (Memory per node in GiB, MiB, KiB) |
--mem=12000mIf not set, the scheduler defaults will be applied. |
| Minimum memory per processor | --mem-per-cpu=<memory> |
--mem-per-cpu=1000mIf not set, the scheduler defaults will be applied. This is the minimum memory required per usable allocated CPU. This will be governed, in part, by the amount of memory available on the node itself. If you need more than 180G of memory per node, we suggest using the largemem partition. |
| Request GPUs |
|
Available types of GPUs: double precision v100, and single precision L40. |
| Process count per GPU | --ntasks-per-gpu=<count>Must be used with --ntasks or --gres=gpu: |
2 tasks per GPU * 4 GPUs = 8 tasks total |
| Job array | --array=<array indices> |
See section below for additional details on using arrays. |
| Standard output file | --output=<file path> (path must exist) |
If not specified, Slurm combines ouput and error into a file named slurm-<jobId>.out. Slurm creates this file by default. This file is potentially useful when troubleshooting your job.
%u = username |
| Standard error file | --error=<file path> (path must exist) |
If not specified, Slurm combines ouput and error into a file named slurm-<jobId>.out. |
| Copy environment | --export=ALL (default)
|
--export=ALL |
| Copy environment variable | --export=<variable=value,var2=val2> |
--export=EDITOR=/bin/vim |
| Job dependency | --dependency=after:jobID[:jobID...]
|
--dependency=after:1234[:1233] |
| Request software license(s) |
Note: to see which software licenses are available run the following command from a login node: |
|
| Request event notification |
Note: multiple mail-type requests may be specified in a comma separated list:
Note: there are multiple mail-type options one can use:
|
|
| Email address | --mail-user=<email address> |
[email protected] |
| Defer job until the specified time | --begin=<date/time> |
--begin=2020-12-25T12:30:00 |