Table of Contents

Batch Job Execution (SLURM)

The VU MIF HPC cluster uses SLURM to manage computing resources and execute jobs.

If you have used a personal computer before, think of SLURM as a workload manager. Instead of running applications directly, you ask SLURM for resources (CPU cores, memory, GPUs, and runtime), and SLURM finds a suitable compute node to run your job.

What Is SLURM?

SLURM is responsible for:

Most HPC work starts with creating a job script and submitting it using:

sbatch job.sh

How Jobs Run

A typical HPC workflow looks like this:

  1. Store your files in the shared Lustre file system
  2. Create a SLURM job script
  3. Submit the job using sbatch
  4. Monitor progress
  5. Review results

<note tip> Unlike a personal computer, computational workloads are normally not executed directly on the login node.

Use SLURM to run calculations on compute nodes. </note>

Available Partitions

The VU MIF HPC cluster currently provides:

Partition Purpose Maximum Runtime
main General-purpose CPU workloads 7 days
gpu GPU-accelerated workloads 48 hours

<note> This guide focuses on the main and gpu partitions, which are used by most users.

The power partition is intended for specialized IBM Power9 workloads and is not covered here. </note>

<note tip> Most users should start with the main partition.

Use the gpu partition only when your application requires NVIDIA GPUs. </note>

<note tip> If no runtime limit is specified, SLURM uses a default runtime limit of 2 hours. </note>

Your First Job Script

Create a file named hello.sh:

nano hello.sh

Paste:

#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --partition=main
#SBATCH --output=hello.out
#SBATCH --time=00:01:00
 
echo "Hello from HPC!"
hostname
date

Submit the job:

sbatch hello.sh

Example output:

Submitted batch job 123456

Common Resource Requests

Many applications require CPU cores, memory, or GPUs.

Example:

#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --partition=main
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=01:00:00
 
python3 script.py

Understanding SLURM Directives

Directive Description
–partition=main Use the CPU partition
–partition=gpu Use the GPU partition
–cpus-per-task=4 Request 4 CPU cores
–mem=8G Request 8 GB RAM
–time=01:00:00 Request 1 hour runtime
–output=file.out Save output to a file
–job-name=myjob Assign a name to the job

Project Allocations

Users working in Waldur-managed projects may need to specify an allocation account:

#SBATCH --account=alloc_xxxx_project

Example:

#!/bin/bash
#SBATCH --account=alloc_xxxx_project
#SBATCH --partition=main
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=01:00:00
 
python3 script.py

<note important> VU MIF users using the default allocation do not need to specify the –account parameter.

Only users assigned to a project allocation should use –account. </note>

Submitting a Job

Submit a job:

sbatch job.sh

SLURM returns a Job ID:

Submitted batch job 123456

The Job ID uniquely identifies your job.

Monitoring Jobs

Show your running and pending jobs:

squeue -u $USER

Example:

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
123456 main test user R 00:10 1 compute07

Why Is My Job Pending?

A job may remain in PENDING state when:

Show detailed status:

squeue -j JOBID

View Completed Jobs

Display information about completed jobs:

sacct -j JOBID

More detailed output:

sacct -j JOBID --format=JobID,State,Elapsed,AllocCPUS

Example:

JobID             State      Elapsed  AllocCPUS
------------ ---------- ------------ ----------
123456       COMPLETED       00:01:05          4

Monitor Resource Usage

While a job is running:

sstat -j JOBID.batch --format=JobID,AveCPU,AveRSS,MaxRSS

Example:

       JobID     AveCPU     AveRSS     MaxRSS
------------ ---------- ---------- ----------
123456.batch   00:00:01      500M      620M

Where:

Field Description
AveCPU Average CPU time consumed by job tasks
AveRSS Average memory usage
MaxRSS Maximum memory usage

Understanding CPU Usage

After a job finishes, you can compare elapsed runtime with total CPU usage:

sacct -j JOBID --format=JobID,AllocCPUS,Elapsed,CPUTime

Example:

JobID          AllocCPUS    Elapsed    CPUTime
------------ ---------- ---------- ----------
123456                4   00:00:10   00:00:38
Field Description
AllocCPUS Number of allocated CPU cores
Elapsed Real (wall-clock) runtime
CPUTime Total CPU time used by all allocated CPU cores

Example: Single-Threaded Application

AllocCPUS = 4
Elapsed   = 00:00:10
CPUTime   = 00:00:10

The application only used approximately one CPU core, even though four cores were allocated.

Example: Multi-Threaded Application

AllocCPUS = 4
Elapsed   = 00:00:10
CPUTime   = 00:00:38

The application actively used multiple CPU cores.

A fully utilized four-core job running for ten seconds would produce approximately:

AllocCPUS = 4
Elapsed   = 00:00:10
CPUTime   = 00:00:40

<note tip> A quick way to verify that your application is using multiple CPU cores is to compare CPUTime with Elapsed.

If CPUTime is several times larger than Elapsed, the workload is utilizing multiple CPU cores concurrently. </note>

Cancelling Jobs

Cancel a running or pending job:

scancel JOBID

Example:

scancel 123456

Interactive Jobs

Sometimes it is useful to work interactively instead of submitting a batch script.

Start an interactive shell:

srun --pty $SHELL

When finished:

exit

GPU Jobs

To reserve GPU resources:

#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=01:00:00
 
nvidia-smi

The parameter –gres=gpu:1 requests one GPU.

The command:

nvidia-smi

displays information about the GPU allocated to the job.

Users working within a project allocation may also need:

#SBATCH --account=alloc_xxxx_project

Useful Commands

Command Purpose
sbatch job.sh Submit a batch job
squeue -u $USER View your jobs
sacct -j JOBID View completed jobs
sstat -j JOBID.batch Monitor running jobs
scancel JOBID Cancel a job
sinfo Show cluster partitions

SLURM Workflow Summary

The typical workflow for running a job is:

  1. Create a job script
  2. Request the required resources (CPU, memory, GPU, runtime)
  3. Submit the job using sbatch
  4. Monitor the job using squeue
  5. Review results and resource usage using sacct or sstat

<note tip> Start with small jobs and modest resource requests. Once your application works correctly, increase CPU, memory, or GPU allocations as needed. </note>

Frequently Asked Questions

Which partition should I use?

Do I need the --account option?

Most VU MIF users do not.

Only users assigned to a project allocation should specify:

#SBATCH --account=alloc_xxxx_project

What happens if I omit --time?

SLURM applies a default runtime limit of 2 hours.

Why is my job pending?

The most common reasons are:

Check:

squeue -j JOBID

Where can I find my job output?

By default:

slurm-JOBID.out

in the directory where the job was submitted.

You can change this using:

#SBATCH --output=myjob.out

Common Problems

Invalid account or account/partition combination specified

Possible causes:

Job Remains Pending

Check:

squeue -j JOBID

Possible causes:

Job Exits Immediately

Check the output file:

cat slurm-JOBID.out

Also review:

sacct -j JOBID

GPU Not Detected

Verify:

Next Steps

Now that you know how to submit and monitor jobs, learn about resource quotas, project allocations, and Waldur-managed projects:

You may also revisit:


<note> This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, software stack, and operational policies. </note>