Skip to content

Update batch computing documentation #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jcurtis2 merged 2 commits into from
Apr 7, 2026
Merged

Update batch computing documentation #31

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
16538bc
improve batch notes. finally add l40s. deprecate gpu
jcurtis2 Apr 7, 2026
473d280
fix gpu details
jcurtis2 Apr 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
259 changes: 183 additions & 76 deletions docs/source/batch.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,68 +8,205 @@ killed if run on the head node.
To avoid this issue, users must access the compute nodes. This can be done in two ways,
interactively or using the batch system which uses `SLURM <https://slurm.schedmd.com/>`__.

Available compute resources
---------------------------

Keeling consists of different partitions that you may submit your job script
to depending on your computing needs. Partitions vary on number of nodes required,
amount of wall time allowed, and availability of special resources
(GPUs or nodes with higher core counts),

+---------+--------------------------------------------+------------------+
| Option | Description | Wall clock limit |
+---------+--------------------------------------------+------------------+
| node | Shared single node | 10 days |
+---------+--------------------------------------------+------------------+
| seseml | Exclusive single node | 7 days |
+---------+--------------------------------------------+------------------+
| sesempi | Multiple node jobs (MinNodes=2 MaxNodes=8) | 7 days |
+---------+--------------------------------------------+------------------+
| sesebig | Larger jobs (MinNodes=9 MaxNodes=32) | 2 days |
+---------+--------------------------------------------+------------------+
| gpu | Access to GPU (Titan Titan RTX), MaxNodes=1 | 2 days |
+---------+--------------------------------------------+------------------+
| L40S | Access to GPU (L40S), MaxNodes=2 | 2 days |
+---------+--------------------------------------------+------------------+

Keeling consists of the following types of nodes with the following features

+------------------------------+----------------+----------------------+----------+
| Name | Cores per node | Real memory (MB) | GPU VRAM |
+------------------------------+----------------+----------------------+----------+
| a | 4 | 12000 | |
+------------------------------+----------------+----------------------+----------+
| b | 8 | 16000, 32000 | |
+------------------------------+----------------+----------------------+----------+
| c | 12 | 32160, 72480, 118000 | |
+------------------------------+----------------+----------------------+----------+
| d | 12 | 64300, 80500 | |
+------------------------------+----------------+----------------------+----------+
| e | 12 | 64300 | |
+------------------------------+----------------+----------------------+----------+
| f | 8 | 24000 | |
+------------------------------+----------------+----------------------+----------+
| g | 20 | 128800 | |
+------------------------------+----------------+----------------------+----------+
| gpu (Titan RTX) | 20 | 253000 | 24 GB |
+------------------------------+----------------+----------------------+----------+
| gpu (L40S) | 96 | 253000 | 46 GB |
+------------------------------+----------------+----------------------+----------+
| h | 20 or 24 | 256200 | |
+------------------------------+----------------+----------------------+----------+
| i | 32 | 192000 | |
+------------------------------+----------------+----------------------+----------+
| j | 48 | 253000 | |
+------------------------------+----------------+----------------------+----------+

Interactive computing
---------------------

An interactive session reserves dedicated resources on compute nodes allowing you to use them
interactively as you would the login node.
To request an interactive job, the ``qlogin`` command is used. An example of a request:

.. code-block:: console

qlogin -n 20 --mem-per-cpu=4096 --time=12:00:00 --job-name=interactive

will utilize 20 cores with 4 GB of memory per core, a wall clock limit of 12 hours.
Note: Interactive job launched by ``qlogin`` are only allowed one node.
will utilize 20 cores with 4 GB of memory per core (specified as 4096 MB),
a wall clock limit of 12 hours.

.. note::

Interactive job launched by ``qlogin`` are only allowed one node.

To finalize your interactive compute job may do:

.. code-block:: console

exit

To finalize your interactive compute job you can do ``exit`` to release
the node to be available to others.
which will end you job and release the node for others to use.

Alternatively you can use ``srun`` to access other partitions to request more resources.

To interactively compute on a GPU node:
Accessing GPU nodes
^^^^^^^^^^^^^^^^^^^

Keeling has 1 Titan RTX GPU node and 8 L40S GPUs. To access these resources, you must specify the
``gpu`` or ``L40S`` partition when requesting an interactive session.

To interactively compute on the Titan RTX GPU node (24 GB VRAM):

.. code-block:: console

qlogin -p gpu -n 20 --gres=gpu:RTX:1 --mem=253000

To interactively compute on a newer L40S GPU node (46 GB VRAM):

.. code-block:: console

qlogin -p gpu -n 20 --gres=gpu:K40m:1
qlogin -p l40s -n 96 --gres=gpu:L40S:1 --mem=253000

.. note::

Once in the interactive sessions, you must run the following
to load the neccessary modules to use the L40S GPU:

.. code-block:: console

module purge
module load L40S

Batch computing
---------------

The login node of Keeling is useful for lightweight computing that requires not a lot of memory
and not a large number of cores. Programs that require large amounts of memory or CPUs will be
killed if run on the head node.
A batch job is controlled by a script written by the user who submits the job to the batch system (Slurm).
The batch system then selects the resources for the job given the parameters from the user
and decides where and when to run the job. This method allows you to have access to more resources.

Jobs are submitted by:

.. code-block:: console

sbatch <job script>

The job script is a shell script that contains the necessary directives for the batch system to
allocate resources for the job. The job script should typically contain the following directives:

+---------------------+-----------------+
| Directive | Description |
+---------------------+-----------------+
| #SBATCH --job-name | Name of the job |
+---------------------+-----------------+
| #SBATCH -p | Partition |
+---------------------+-----------------+
| #SBATCH -n | Number of cores |
+---------------------+-----------------+
| #SBATCH --time | Wall clock time |
+---------------------+-----------------+
| #SBATCH --mem | Memory |
+---------------------+-----------------+

For GPU batch computing:

+----------------+----------------------------------------------------+
| Directive | Description |
+----------------+----------------------------------------------------+
| #SBATCH --gres | Resources request (GPUS). For L40S, ``gpu:L40S:1`` |
+----------------+----------------------------------------------------+


Additional useful directives:

+-----------------------+----------------------------------------------+
| Directive | Description |
+-----------------------+----------------------------------------------+
| #SBATCH --mail-user | Email address to receive notifications |
+-----------------------+----------------------------------------------+
| #SBATCH --mail-type | Reasons to notify (e.g. BEGIN, END, FAIL) |
+-----------------------+----------------------------------------------+
| #SBATCH --constraint | Node constraint (i.e. required node feature) |
+-----------------------+----------------------------------------------+
| #SBATCH --sockets | Number of sockets per node |
+-----------------------+----------------------------------------------+
| #SBATCH --cores | Number of cores per socket |
+-----------------------+----------------------------------------------+
| #SBATCH --mem-per-cpu | Memory per core. Alternative to --mem |
+-----------------------+----------------------------------------------+

It can also be used to launch interactive or non-interactive batch jobs using the cluster鈥檚 scheduler slurm. Keeling is a shared resource and is monitored for abuse. CPU or memory intensive jobs will be automatically killed - these should be run on compute nodes to maintain system stability.

Example scripts
^^^^^^^^^^^^^^^

A simple script requesting a single node consisting of 12 cores for 24 hours:

.. code-block:: slurm

#!/bin/bash

#SBATCH --job-name=PARTMC3D
#SBATCH --job-name=<job name>
#SBATCH -p node
#SBATCH -n 12
#SBATCH -n 12
#SBATCH --time=24:00:00
#SBATCH --mail-type=FAIL
#SBATCH --mail-type=END
#SBATCH --mail-user=jcurtis2@illinois.edu
#SBATCH --mail-user=<netID>@illinois.edu

A simple script requesting many nodes consisting of 100 cores for 12 hours
A simple script requesting many nodes consisting of 100 cores for 12 hours
without caring about node configuration:

.. code-block:: slurm

#!/bin/bash

#SBATCH --job-name=PARTMC3D
#SBATCH -p sesempi
#SBATCH -n 100
#SBATCH --job-name=<job name>
#SBATCH -p sesempi
#SBATCH -n 100
#SBATCH --time=12:00:00
#SBATCH --mail-type=FAIL
#SBATCH --mail-type=END
#SBATCH --mail-user=jcurtis2@illinois.edu
#SBATCH --mail-user=<netID>@illinois.edu

A simple script for requesting a handful of cores but a large amount of memory per
core (16 GB) for 6 hours:
Expand All @@ -78,34 +215,52 @@ core (16 GB) for 6 hours:

#!/bin/bash

#SBATCH --job-name=PARTMC3D
#SBATCH -p node
#SBATCH --job-name=<job name>
#SBATCH -p node
#SBATCH -n 4
#SBATCH --time=12:00:00
#SBATCH --mem-per-cpu=16g

A more advanced job script requesting 100 cores, 4 GB of memory per core, 12 hours of
wall clock time and wanting a node configuration of 2 sockets with 10 cores each:
wall clock time and wanting a node configuration of 2 sockets with 10 cores each
(i.e. using exclusively keeling-g nodes) and running WRF:

.. code-block:: slurm

#!/bin/bash

#SBATCH --job-name=PARTMC3D
#SBATCH --job-name=<job name>
#SBATCH -p sesempi
#SBATCH -n 100
#SBATCH -n 100
#SBATCH --sockets-per-node=2
#SBATCH --cores-per-socket=10
#SBATCH --time=12:00:00
#SBATCH --mem-per-cpu=4096
#SBATCH --mail-type=FAIL
#SBATCH --mail-type=END
#SBATCH --mail-user=jcurtis2@illinois.edu
#SBATCH --mail-user=<netID>@illinois.edu

cd WRFV3/test/em_real/
mpirun -np $SLURM_NTASKS ./wrf.exe

Helpful SLURM command line options:
A script for accessing a L40S GPU, requesting 96 cores, 2 days of wall clock time, and 253 GB of memory:

.. code-block:: slurm

#SBATCH -p l40s
#SBATCH -N 1
#SBATCH -n 96
#SBATCH --gres=gpu:L40S:1
#SBATCH --constraint=tl40s
#SBATCH --mem=253000
#SBATCH --time=48:00:00
#SBATCH --output=batchout
#SBATCH --mail-user=<netID>@illinois.edu

module purge
module load L40S

Helpful SLURM command line options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

+-------------+-------------------------------------------------------------------+
| Command | Description |
Expand All @@ -129,59 +284,11 @@ Helpful SLURM command line options:
Information regarding each command may be found `here <https://slurm.schedmd.com/sinfo.html>`_
or by viewing each command's ``man`` page.

Partitions
______________

Keeling consists of many different partitions that you may submit your job script
to depending on your computing needs.

+-------------+------------------------+--------------------+
| Option | Description | Max wall clock time|
+-------------+------------------------+--------------------+
| node | Shared single node | 10 days |
+-------------+------------------------+--------------------+
| seseml | Exclusive single node | 7 days |
+-------------+------------------------+--------------------+
| sesempi | MinNodes=2 MaxNodes=8 | 7 days |
+-------------+------------------------+--------------------+
| sesebig | MinNodes=9 MaxNodes=32 | 2 days |
+-------------+------------------------+--------------------+
| gpu | Access to GPU (K40m) | 7 days |
+-------------+------------------------+--------------------+

Keeling consists of the following types of nodes with the following features

+-------------+------------------+---------------+
| Name | Cores per node | Real memory |
+-------------+------------------+---------------+
| a | 4 | 12000 |
+-------------+------------------+---------------+
| b | 8 | 16000 - 32000 |
+-------------+------------------+---------------+
| c | 12 | 32160 or 72480|
+-------------+------------------+---------------+
| d | 12 | 64300 or 80500|
+-------------+------------------+---------------+
| e | 12 | 64300 |
+-------------+------------------+---------------+
| f | 8 | 24000 |
+-------------+------------------+---------------+
| g | 20 | 128800 |
+-------------+------------------+---------------+
| gpu | 20 | 256200 |
+-------------+------------------+---------------+
| h | 20 or 24 | 256200 |
+-------------+------------------+---------------+
| i | 32 | 192000 |
+-------------+------------------+---------------+
| j | 48 | 253000 |
+-------------+------------------+---------------+

Access compute nodes directly using SSH
---------------------------------------

This is not allowed except for monitoring already running jobs. However if you
need to monitor a job, you may access the specific compute node by first
need to monitor a job (or set up SSH tunnel), you may access the specific compute node by first
identifying the node your job is running on by:

.. code-block:: console
Expand All @@ -192,6 +299,6 @@ which will list information regarding the your running and queued jobs with ``NO
denoting the nodes of your running jobs.
You may then ``ssh`` directly into that node by the following:

.. code-block:: console
.. code-block:: console

ssh keeling-<node letter and number>
Loading