This is an old revision of the document!
Table of Contents
Batch Job Execution (SLURM)
The VU MIF HPC cluster uses SLURM to manage computing resources and execute jobs.
If you have used a personal computer before, think of SLURM as a workload manager. Instead of running applications directly, you ask SLURM for resources (CPU cores, memory, GPUs, and runtime) and SLURM finds a suitable compute node to run your job.
What Is SLURM?
SLURM is responsible for:
- Allocating cluster resources
- Starting and stopping jobs
- Managing queues
- Enforcing resource limits
- Tracking resource usage
Most HPC work starts with creating a job script and submitting it using:
sbatch job.sh
How Jobs Run
A typical HPC workflow looks like this:
- Prepare your files in Lustre storage
- Create a SLURM job script
- Submit the job using
sbatch - Monitor progress
- Review results
<note tip> Unlike a personal computer, computational workloads are normally not executed directly on the login node.
Use SLURM to run calculations on compute nodes. </note>
Available Partitions
The VU MIF HPC cluster currently provides:
| Partition | Purpose | Maximum Runtime |
|---|---|---|
main | General-purpose CPU workloads | 7 days |
gpu | GPU-accelerated workloads | 48 hours |
<note tip>
Most users should start with the main partition.
Use the gpu partition only when your application requires NVIDIA GPUs.
</note>
<note tip> If no runtime limit is specified, SLURM uses a default runtime limit of 2 hours. </note>
Your First Job Script
Create a file named hello.sh:
nano hello.sh
Paste:
#!/bin/bash #SBATCH --job-name=hello #SBATCH --partition=main #SBATCH --output=hello.out #SBATCH --time=00:01:00 echo "Hello from HPC!" hostname date
Submit the job:
sbatch hello.sh
Example output:
Submitted batch job 123456
Common Resource Requests
Many applications require CPU cores, memory, or GPUs.
Example:
#!/bin/bash #SBATCH --job-name=myjob #SBATCH --partition=main #SBATCH --cpus-per-task=4 #SBATCH --mem=8G #SBATCH --time=01:00:00 python3 script.py
Understanding SLURM Directives
| Directive | Description |
|---|---|
–partition=main | Use the CPU partition |
–partition=gpu | Use the GPU partition |
–cpus-per-task=4 | Request 4 CPU cores |
–mem=8G | Request 8 GB RAM |
–time=01:00:00 | Request 1 hour runtime |
–output=file.out | Save output to a file |
–job-name=myjob | Assign a name to the job |
Project Allocations
Users working in Waldur-managed projects may need to specify an allocation account:
#SBATCH --account=alloc_xxxx_project
Example:
#!/bin/bash #SBATCH --account=alloc_xxxx_project #SBATCH --partition=main #SBATCH --cpus-per-task=4 #SBATCH --mem=8G #SBATCH --time=01:00:00 python3 script.py
<note important>
VU MIF users using the default allocation do not need to specify the –account parameter.
Only users assigned to a project allocation should use –account.
</note>
Submitting a Job
Submit a job:
sbatch job.sh
SLURM returns a Job ID:
Submitted batch job 123456
The Job ID uniquely identifies your job.
Monitoring Jobs
Show your running and pending jobs:
squeue -u $USER
Example:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 123456 main test user R 00:10 1 compute07
Why Is My Job Pending?
A job may remain in PENDING state when:
- Requested resources are currently unavailable
- Higher-priority jobs are running
- Resource limits have been reached
- An incorrect partition or allocation was specified
Show detailed status:
squeue -j JOBID
View Completed Jobs
Display information about completed jobs:
sacct -j JOBID
More detailed output:
sacct -j JOBID --format=JobID,State,Elapsed,AllocCPUS
Example:
JobID State Elapsed AllocCPUS ------------ ---------- ---------- ---------- 123456 COMPLETED 00:01:05 4
Monitor Resource Usage
While a job is running:
sstat -j JOBID.batch --format=JobID,AveCPU,AveRSS,MaxRSS
Example:
JobID AveCPU AveRSS MaxRSS ------------ ---------- ---------- ---------- 123456.batch 00:00:01 500M 620M
Cancelling Jobs
Cancel a running or pending job:
scancel JOBID
Example:
scancel 123456
Interactive Jobs
Sometimes it is useful to work interactively instead of submitting a batch script.
Start an interactive shell:
srun --pty $SHELL
When finished:
exit
GPU Jobs
To reserve GPU resources:
#!/bin/bash #SBATCH --partition=gpu #SBATCH --gres=gpu:1 #SBATCH --cpus-per-task=4 #SBATCH --mem=16G #SBATCH --time=01:00:00 nvidia-smi
The parameter:
--gres=gpu:1
requests one GPU.
Interactive GPU Session
Start an interactive GPU session:
srun -p gpu --gres=gpu:1 --pty $SHELL
Verify GPU availability:
nvidia-smi
MPI Jobs
MPI applications can span multiple CPU cores or nodes.
Example:
#!/bin/bash #SBATCH --partition=main #SBATCH -n 4 module load openmpi mpirun my_application
Available OpenMPI versions:
module avail openmpi
Load a version:
module load openmpi/4.0
Task Efficiency
Efficient jobs help everyone use the cluster effectively.
Good practice:
- Request only the resources you need
- Use at least 50% of allocated CPU resources
- Match application thread count to requested CPUs
- Set a realistic runtime limit
Avoid:
- Requesting excessive CPUs
- Requesting excessive memory
- Reserving GPUs that are not used
- Running large workloads on login nodes
Common Problems
Invalid account or account/partition combination specified
Possible causes:
- Your SLURM account is not yet created
- Your allocation does not exist
- You do not have access to the specified allocation
Job Remains Pending
Check:
squeue -j JOBID
Possible causes:
- Resources unavailable
- Partition unavailable
- Resource limits exceeded
Job Exits Immediately
Check the output file:
cat slurm-JOBID.out
Also review accounting information:
sacct -j JOBID
GPU Not Detected
Verify:
–partition=gpuis used–gres=gpu:Nis specified- GPU resources are available
Useful Commands
| Command | Purpose |
|---|---|
sbatch job.sh | Submit a batch job |
squeue -u $USER | View your jobs |
sacct -j JOBID | View completed jobs |
sstat -j JOBID.batch | Monitor running jobs |
scancel JOBID | Cancel a job |
sinfo | Show cluster partitions |
Next Steps
Now that you know how to submit jobs, learn how resource allocations and project quotas work:
You may also want to revisit:
<note> This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, software stack, and operational policies. </note>
