IT wiki

VU MIF STSC

User Tools

Site Tools


en:hpc-new:slurm

This is an old revision of the document!


Batch Job Execution (SLURM)

The VU MIF HPC cluster uses SLURM to manage computing resources and execute jobs.

If you have used a personal computer before, think of SLURM as a workload manager. Instead of running applications directly, you ask SLURM for resources (CPU cores, memory, GPUs, and runtime) and SLURM finds a suitable compute node to run your job.

What Is SLURM?

SLURM is responsible for:

  • Allocating cluster resources
  • Starting and stopping jobs
  • Managing queues
  • Enforcing resource limits
  • Tracking resource usage

Most HPC work starts with creating a job script and submitting it using:

sbatch job.sh

How Jobs Run

A typical HPC workflow looks like this:

  1. Prepare your files in Lustre storage
  2. Create a SLURM job script
  3. Submit the job using sbatch
  4. Monitor progress
  5. Review results

<note tip> Unlike a personal computer, computational workloads are normally not executed directly on the login node.

Use SLURM to run calculations on compute nodes. </note>

Available Partitions

The VU MIF HPC cluster currently provides:

Partition Purpose Maximum Runtime
main General-purpose CPU workloads 7 days
gpu GPU-accelerated workloads 48 hours

<note tip> Most users should start with the main partition.

Use the gpu partition only when your application requires NVIDIA GPUs. </note>

<note tip> If no runtime limit is specified, SLURM uses a default runtime limit of 2 hours. </note>

Your First Job Script

Create a file named hello.sh:

nano hello.sh

Paste:

#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --partition=main
#SBATCH --output=hello.out
#SBATCH --time=00:01:00
 
echo "Hello from HPC!"
hostname
date

Submit the job:

sbatch hello.sh

Example output:

Submitted batch job 123456

Common Resource Requests

Many applications require CPU cores, memory, or GPUs.

Example:

#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --partition=main
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=01:00:00
 
python3 script.py

Understanding SLURM Directives

Directive Description
–partition=main Use the CPU partition
–partition=gpu Use the GPU partition
–cpus-per-task=4 Request 4 CPU cores
–mem=8G Request 8 GB RAM
–time=01:00:00 Request 1 hour runtime
–output=file.out Save output to a file
–job-name=myjob Assign a name to the job

Project Allocations

Users working in Waldur-managed projects may need to specify an allocation account:

#SBATCH --account=alloc_xxxx_project

Example:

#!/bin/bash
#SBATCH --account=alloc_xxxx_project
#SBATCH --partition=main
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=01:00:00
 
python3 script.py

<note important> VU MIF users using the default allocation do not need to specify the –account parameter.

Only users assigned to a project allocation should use –account. </note>

Submitting a Job

Submit a job:

sbatch job.sh

SLURM returns a Job ID:

Submitted batch job 123456

The Job ID uniquely identifies your job.

Monitoring Jobs

Show your running and pending jobs:

squeue -u $USER

Example:

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
123456 main test user R 00:10 1 compute07

Why Is My Job Pending?

A job may remain in PENDING state when:

  • Requested resources are currently unavailable
  • Higher-priority jobs are running
  • Resource limits have been reached
  • An incorrect partition or allocation was specified

Show detailed status:

squeue -j JOBID

View Completed Jobs

Display information about completed jobs:

sacct -j JOBID

More detailed output:

sacct -j JOBID --format=JobID,State,Elapsed,AllocCPUS

Example:

JobID             State    Elapsed  AllocCPUS
------------ ---------- ---------- ----------
123456       COMPLETED    00:01:05          4

Monitor Resource Usage

While a job is running:

sstat -j JOBID.batch --format=JobID,AveCPU,AveRSS,MaxRSS

Example:

       JobID     AveCPU     AveRSS     MaxRSS
------------ ---------- ---------- ----------
123456.batch   00:00:01      500M      620M

Cancelling Jobs

Cancel a running or pending job:

scancel JOBID

Example:

scancel 123456

Interactive Jobs

Sometimes it is useful to work interactively instead of submitting a batch script.

Start an interactive shell:

srun --pty $SHELL

When finished:

exit

GPU Jobs

To reserve GPU resources:

#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=01:00:00
 
nvidia-smi

The parameter:

--gres=gpu:1

requests one GPU.

Interactive GPU Session

Start an interactive GPU session:

srun -p gpu --gres=gpu:1 --pty $SHELL

Verify GPU availability:

nvidia-smi

MPI Jobs

MPI applications can span multiple CPU cores or nodes.

Example:

#!/bin/bash
#SBATCH --partition=main
#SBATCH -n 4
 
module load openmpi
 
mpirun my_application

Available OpenMPI versions:

module avail openmpi

Load a version:

module load openmpi/4.0

Task Efficiency

Efficient jobs help everyone use the cluster effectively.

Good practice:

  • Request only the resources you need
  • Use at least 50% of allocated CPU resources
  • Match application thread count to requested CPUs
  • Set a realistic runtime limit

Avoid:

  • Requesting excessive CPUs
  • Requesting excessive memory
  • Reserving GPUs that are not used
  • Running large workloads on login nodes

Common Problems

Invalid account or account/partition combination specified

Possible causes:

  • Your SLURM account is not yet created
  • Your allocation does not exist
  • You do not have access to the specified allocation

Job Remains Pending

Check:

squeue -j JOBID

Possible causes:

  • Resources unavailable
  • Partition unavailable
  • Resource limits exceeded

Job Exits Immediately

Check the output file:

cat slurm-JOBID.out

Also review accounting information:

sacct -j JOBID

GPU Not Detected

Verify:

  • –partition=gpu is used
  • –gres=gpu:N is specified
  • GPU resources are available

Useful Commands

Command Purpose
sbatch job.sh Submit a batch job
squeue -u $USER View your jobs
sacct -j JOBID View completed jobs
sstat -j JOBID.batch Monitor running jobs
scancel JOBID Cancel a job
sinfo Show cluster partitions

Next Steps

Now that you know how to submit jobs, learn how resource allocations and project quotas work:

You may also want to revisit:


<note> This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, software stack, and operational policies. </note>

en/hpc-new/slurm.1795099599.txt.gz · Last modified: by eduardas

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki