Table of Contents
Batch Job Execution (SLURM)
The VU MIF HPC cluster uses SLURM to manage computing resources and execute jobs.
If you have used a personal computer before, think of SLURM as a workload manager. Instead of running applications directly, you ask SLURM for resources (CPU cores, memory, GPUs, and runtime), and SLURM finds a suitable compute node to run your job.
What Is SLURM?
SLURM is responsible for:
- Allocating cluster resources
- Starting and stopping jobs
- Managing queues
- Enforcing resource limits
- Tracking resource usage
Most HPC work starts with creating a job script and submitting it using:
sbatch job.sh
How Jobs Run
A typical HPC workflow looks like this:
- Store your files in the shared Lustre file system
- Create a SLURM job script
- Submit the job using
sbatch - Monitor progress
- Review results
<note tip> Unlike a personal computer, computational workloads are normally not executed directly on the login node.
Use SLURM to run calculations on compute nodes. </note>
Available Partitions
The VU MIF HPC cluster currently provides:
| Partition | Purpose | Maximum Runtime |
|---|---|---|
main | General-purpose CPU workloads | 7 days |
gpu | GPU-accelerated workloads | 48 hours |
<note>
This guide focuses on the main and gpu partitions, which are used by most users.
The power partition is intended for specialized IBM Power9 workloads and is not covered here.
</note>
<note tip>
Most users should start with the main partition.
Use the gpu partition only when your application requires NVIDIA GPUs.
</note>
<note tip> If no runtime limit is specified, SLURM uses a default runtime limit of 2 hours. </note>
Your First Job Script
Create a file named hello.sh:
nano hello.sh
Paste:
#!/bin/bash #SBATCH --job-name=hello #SBATCH --partition=main #SBATCH --output=hello.out #SBATCH --time=00:01:00 echo "Hello from HPC!" hostname date
Submit the job:
sbatch hello.sh
Example output:
Submitted batch job 123456
Common Resource Requests
Many applications require CPU cores, memory, or GPUs.
Example:
#!/bin/bash #SBATCH --job-name=myjob #SBATCH --partition=main #SBATCH --cpus-per-task=4 #SBATCH --mem=8G #SBATCH --time=01:00:00 python3 script.py
Understanding SLURM Directives
| Directive | Description |
|---|---|
–partition=main | Use the CPU partition |
–partition=gpu | Use the GPU partition |
–cpus-per-task=4 | Request 4 CPU cores |
–mem=8G | Request 8 GB RAM |
–time=01:00:00 | Request 1 hour runtime |
–output=file.out | Save output to a file |
–job-name=myjob | Assign a name to the job |
Project Allocations
Users working in Waldur-managed projects may need to specify an allocation account:
#SBATCH --account=alloc_xxxx_project
Example:
#!/bin/bash #SBATCH --account=alloc_xxxx_project #SBATCH --partition=main #SBATCH --cpus-per-task=4 #SBATCH --mem=8G #SBATCH --time=01:00:00 python3 script.py
<note important>
VU MIF users using the default allocation do not need to specify the –account parameter.
Only users assigned to a project allocation should use –account.
</note>
Submitting a Job
Submit a job:
sbatch job.sh
SLURM returns a Job ID:
Submitted batch job 123456
The Job ID uniquely identifies your job.
Monitoring Jobs
Show your running and pending jobs:
squeue -u $USER
Example:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 123456 main test user R 00:10 1 compute07
Why Is My Job Pending?
A job may remain in PENDING state when:
- Requested resources are currently unavailable
- Higher-priority jobs are running
- Resource limits have been reached
- An incorrect partition or allocation was specified
Show detailed status:
squeue -j JOBID
View Completed Jobs
Display information about completed jobs:
sacct -j JOBID
More detailed output:
sacct -j JOBID --format=JobID,State,Elapsed,AllocCPUS
Example:
JobID State Elapsed AllocCPUS ------------ ---------- ------------ ---------- 123456 COMPLETED 00:01:05 4
Monitor Resource Usage
While a job is running:
sstat -j JOBID.batch --format=JobID,AveCPU,AveRSS,MaxRSS
Example:
JobID AveCPU AveRSS MaxRSS ------------ ---------- ---------- ---------- 123456.batch 00:00:01 500M 620M
Where:
| Field | Description |
|---|---|
AveCPU | Average CPU time consumed by job tasks |
AveRSS | Average memory usage |
MaxRSS | Maximum memory usage |
Understanding CPU Usage
After a job finishes, you can compare elapsed runtime with total CPU usage:
sacct -j JOBID --format=JobID,AllocCPUS,Elapsed,CPUTime
Example:
JobID AllocCPUS Elapsed CPUTime ------------ ---------- ---------- ---------- 123456 4 00:00:10 00:00:38
| Field | Description |
|---|---|
AllocCPUS | Number of allocated CPU cores |
Elapsed | Real (wall-clock) runtime |
CPUTime | Total CPU time used by all allocated CPU cores |
Example: Single-Threaded Application
AllocCPUS = 4 Elapsed = 00:00:10 CPUTime = 00:00:10
The application only used approximately one CPU core, even though four cores were allocated.
Example: Multi-Threaded Application
AllocCPUS = 4 Elapsed = 00:00:10 CPUTime = 00:00:38
The application actively used multiple CPU cores.
A fully utilized four-core job running for ten seconds would produce approximately:
AllocCPUS = 4 Elapsed = 00:00:10 CPUTime = 00:00:40
<note tip>
A quick way to verify that your application is using multiple CPU cores is to compare CPUTime with Elapsed.
If CPUTime is several times larger than Elapsed, the workload is utilizing multiple CPU cores concurrently.
</note>
Cancelling Jobs
Cancel a running or pending job:
scancel JOBID
Example:
scancel 123456
Interactive Jobs
Sometimes it is useful to work interactively instead of submitting a batch script.
Start an interactive shell:
srun --pty $SHELL
When finished:
exit
GPU Jobs
To reserve GPU resources:
#!/bin/bash #SBATCH --partition=gpu #SBATCH --gres=gpu:1 #SBATCH --cpus-per-task=4 #SBATCH --mem=16G #SBATCH --time=01:00:00 nvidia-smi
The parameter –gres=gpu:1 requests one GPU.
The command:
nvidia-smi
displays information about the GPU allocated to the job.
Users working within a project allocation may also need:
#SBATCH --account=alloc_xxxx_project
Useful Commands
| Command | Purpose |
|---|---|
sbatch job.sh | Submit a batch job |
squeue -u $USER | View your jobs |
sacct -j JOBID | View completed jobs |
sstat -j JOBID.batch | Monitor running jobs |
scancel JOBID | Cancel a job |
sinfo | Show cluster partitions |
SLURM Workflow Summary
The typical workflow for running a job is:
- Create a job script
- Request the required resources (CPU, memory, GPU, runtime)
- Submit the job using
sbatch - Monitor the job using
squeue - Review results and resource usage using
sacctorsstat
<note tip> Start with small jobs and modest resource requests. Once your application works correctly, increase CPU, memory, or GPU allocations as needed. </note>
Frequently Asked Questions
Which partition should I use?
- Use
mainfor most workloads. - Use
gpuonly when your application requires NVIDIA GPUs.
Do I need the --account option?
Most VU MIF users do not.
Only users assigned to a project allocation should specify:
#SBATCH --account=alloc_xxxx_project
What happens if I omit --time?
SLURM applies a default runtime limit of 2 hours.
Why is my job pending?
The most common reasons are:
- Insufficient available resources
- Incorrect partition selection
- Resource limits reached
- Invalid allocation account
Check:
squeue -j JOBID
Where can I find my job output?
By default:
slurm-JOBID.out
in the directory where the job was submitted.
You can change this using:
#SBATCH --output=myjob.out
Common Problems
Invalid account or account/partition combination specified
Possible causes:
- Your SLURM account is not yet created
- The allocation name does not exist
- You do not have access to the specified allocation
Job Remains Pending
Check:
squeue -j JOBID
Possible causes:
- Resources unavailable
- Incorrect partition
- Resource limits exceeded
Job Exits Immediately
Check the output file:
cat slurm-JOBID.out
Also review:
sacct -j JOBID
GPU Not Detected
Verify:
–partition=gpuis used–gres=gpu:Nis specified- GPU resources are available
- Your allocation includes GPU access
Next Steps
Now that you know how to submit and monitor jobs, learn about resource quotas, project allocations, and Waldur-managed projects:
You may also revisit:
<note> This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, software stack, and operational policies. </note>
