en:hpc-new:slurm
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revision | |||
| en:hpc-new:slurm [2026/11/19 14:46] – eduardas | en:hpc-new:slurm [2026/11/19 14:51] (current) – eduardas | ||
|---|---|---|---|
| Line 3: | Line 3: | ||
| The VU MIF HPC cluster uses SLURM to manage computing resources and execute jobs. | The VU MIF HPC cluster uses SLURM to manage computing resources and execute jobs. | ||
| - | If you have used a personal computer before, think of SLURM as a workload manager. Instead of running applications directly, you ask SLURM for resources (CPU cores, memory, GPUs, and runtime) and SLURM finds a suitable compute node to run your job. | + | If you have used a personal computer before, think of SLURM as a workload manager. Instead of running applications directly, you ask SLURM for resources (CPU cores, memory, GPUs, and runtime), and SLURM finds a suitable compute node to run your job. |
| ===== What Is SLURM? ===== | ===== What Is SLURM? ===== | ||
| Line 17: | Line 17: | ||
| Most HPC work starts with creating a job script and submitting it using: | Most HPC work starts with creating a job script and submitting it using: | ||
| - | < | + | < |
| sbatch job.sh | sbatch job.sh | ||
| </ | </ | ||
| Line 25: | Line 25: | ||
| A typical HPC workflow looks like this: | A typical HPC workflow looks like this: | ||
| - | - Prepare | + | - Store your files in the shared |
| - Create a SLURM job script | - Create a SLURM job script | ||
| - Submit the job using '' | - Submit the job using '' | ||
| Line 44: | Line 44: | ||
| | '' | | '' | ||
| | '' | | '' | ||
| + | |||
| + | < | ||
| + | This guide focuses on the '' | ||
| + | |||
| + | The '' | ||
| + | </ | ||
| <note tip> | <note tip> | ||
| Line 59: | Line 65: | ||
| Create a file named '' | Create a file named '' | ||
| - | < | + | < |
| nano hello.sh | nano hello.sh | ||
| </ | </ | ||
| Line 65: | Line 71: | ||
| Paste: | Paste: | ||
| - | < | + | < |
| #!/bin/bash | #!/bin/bash | ||
| #SBATCH --job-name=hello | #SBATCH --job-name=hello | ||
| Line 79: | Line 85: | ||
| Submit the job: | Submit the job: | ||
| - | < | + | < |
| sbatch hello.sh | sbatch hello.sh | ||
| </ | </ | ||
| Line 85: | Line 91: | ||
| Example output: | Example output: | ||
| - | < | + | < |
| Submitted batch job 123456 | Submitted batch job 123456 | ||
| </ | </ | ||
| Line 95: | Line 101: | ||
| Example: | Example: | ||
| - | < | + | < |
| #!/bin/bash | #!/bin/bash | ||
| #SBATCH --job-name=myjob | #SBATCH --job-name=myjob | ||
| Line 121: | Line 127: | ||
| Users working in Waldur-managed projects may need to specify an allocation account: | Users working in Waldur-managed projects may need to specify an allocation account: | ||
| - | < | + | < |
| #SBATCH --account=alloc_xxxx_project | #SBATCH --account=alloc_xxxx_project | ||
| </ | </ | ||
| Line 127: | Line 133: | ||
| Example: | Example: | ||
| - | < | + | < |
| #!/bin/bash | #!/bin/bash | ||
| #SBATCH --account=alloc_xxxx_project | #SBATCH --account=alloc_xxxx_project | ||
| Line 148: | Line 154: | ||
| Submit a job: | Submit a job: | ||
| - | < | + | < |
| sbatch job.sh | sbatch job.sh | ||
| </ | </ | ||
| Line 154: | Line 160: | ||
| SLURM returns a Job ID: | SLURM returns a Job ID: | ||
| - | < | + | < |
| Submitted batch job 123456 | Submitted batch job 123456 | ||
| </ | </ | ||
| Line 164: | Line 170: | ||
| Show your running and pending jobs: | Show your running and pending jobs: | ||
| - | < | + | < |
| squeue -u $USER | squeue -u $USER | ||
| </ | </ | ||
| Line 170: | Line 176: | ||
| Example: | Example: | ||
| - | < | + | < |
| JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) | JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) | ||
| 123456 main test user R 00:10 1 compute07 | 123456 main test user R 00:10 1 compute07 | ||
| Line 186: | Line 192: | ||
| Show detailed status: | Show detailed status: | ||
| - | < | + | < |
| squeue -j JOBID | squeue -j JOBID | ||
| </ | </ | ||
| Line 194: | Line 200: | ||
| Display information about completed jobs: | Display information about completed jobs: | ||
| - | < | + | < |
| sacct -j JOBID | sacct -j JOBID | ||
| </ | </ | ||
| Line 200: | Line 206: | ||
| More detailed output: | More detailed output: | ||
| - | < | + | < |
| sacct -j JOBID --format=JobID, | sacct -j JOBID --format=JobID, | ||
| </ | </ | ||
| Line 206: | Line 212: | ||
| Example: | Example: | ||
| - | < | + | < |
| - | JobID | + | JobID |
| - | ------------ ---------- ---------- ---------- | + | ------------ ---------- ------------ ---------- |
| - | 123456 | + | 123456 |
| </ | </ | ||
| Line 216: | Line 222: | ||
| While a job is running: | While a job is running: | ||
| - | < | + | < |
| sstat -j JOBID.batch --format=JobID, | sstat -j JOBID.batch --format=JobID, | ||
| </ | </ | ||
| Line 222: | Line 228: | ||
| Example: | Example: | ||
| - | < | + | < |
| | | ||
| ------------ ---------- ---------- ---------- | ------------ ---------- ---------- ---------- | ||
| 123456.batch | 123456.batch | ||
| </ | </ | ||
| + | |||
| + | Where: | ||
| + | |||
| + | ^ Field ^ Description ^ | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | |||
| + | ===== Understanding CPU Usage ===== | ||
| + | |||
| + | After a job finishes, you can compare elapsed runtime with total CPU usage: | ||
| + | |||
| + | <code shell> | ||
| + | sacct -j JOBID --format=JobID, | ||
| + | </ | ||
| + | |||
| + | Example: | ||
| + | |||
| + | <code text> | ||
| + | JobID AllocCPUS | ||
| + | ------------ ---------- ---------- ---------- | ||
| + | 123456 | ||
| + | </ | ||
| + | |||
| + | ^ Field ^ Description ^ | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | |||
| + | ==== Example: Single-Threaded Application ==== | ||
| + | |||
| + | <code text> | ||
| + | AllocCPUS = 4 | ||
| + | Elapsed | ||
| + | CPUTime | ||
| + | </ | ||
| + | |||
| + | The application only used approximately one CPU core, even though four cores were allocated. | ||
| + | |||
| + | ==== Example: Multi-Threaded Application ==== | ||
| + | |||
| + | <code text> | ||
| + | AllocCPUS = 4 | ||
| + | Elapsed | ||
| + | CPUTime | ||
| + | </ | ||
| + | |||
| + | The application actively used multiple CPU cores. | ||
| + | |||
| + | A fully utilized four-core job running for ten seconds would produce approximately: | ||
| + | |||
| + | <code text> | ||
| + | AllocCPUS = 4 | ||
| + | Elapsed | ||
| + | CPUTime | ||
| + | </ | ||
| + | |||
| + | <note tip> | ||
| + | A quick way to verify that your application is using multiple CPU cores is to compare '' | ||
| + | |||
| + | If '' | ||
| + | </ | ||
| ===== Cancelling Jobs ===== | ===== Cancelling Jobs ===== | ||
| Line 232: | Line 300: | ||
| Cancel a running or pending job: | Cancel a running or pending job: | ||
| - | < | + | < |
| scancel JOBID | scancel JOBID | ||
| </ | </ | ||
| Line 238: | Line 306: | ||
| Example: | Example: | ||
| - | < | + | < |
| scancel 123456 | scancel 123456 | ||
| </ | </ | ||
| Line 248: | Line 316: | ||
| Start an interactive shell: | Start an interactive shell: | ||
| - | < | + | < |
| srun --pty $SHELL | srun --pty $SHELL | ||
| </ | </ | ||
| Line 254: | Line 322: | ||
| When finished: | When finished: | ||
| - | < | + | < |
| exit | exit | ||
| </ | </ | ||
| Line 262: | Line 330: | ||
| To reserve GPU resources: | To reserve GPU resources: | ||
| - | < | + | < |
| #!/bin/bash | #!/bin/bash | ||
| #SBATCH --partition=gpu | #SBATCH --partition=gpu | ||
| Line 273: | Line 341: | ||
| </ | </ | ||
| - | The parameter: | + | The parameter |
| - | < | + | The command: |
| - | --gres=gpu:1 | + | |
| + | < | ||
| + | nvidia-smi | ||
| </ | </ | ||
| - | requests one GPU. | + | displays information about the GPU allocated to the job. |
| - | ===== Interactive GPU Session ===== | + | Users working within a project allocation may also need: |
| - | Start an interactive GPU session: | + | < |
| - | + | # | |
| - | < | + | |
| - | srun -p gpu --gres=gpu:1 --pty $SHELL | + | |
| </ | </ | ||
| - | Verify GPU availability: | + | ===== Useful Commands ===== |
| - | <code bash> | + | ^ Command ^ Purpose ^ |
| - | nvidia-smi | + | | '' |
| - | </ | + | | '' |
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| - | ===== MPI Jobs ===== | + | ===== SLURM Workflow Summary |
| - | MPI applications can span multiple CPU cores or nodes. | + | The typical workflow for running a job is: |
| - | Example: | + | - Create a job script |
| + | - Request the required resources (CPU, memory, GPU, runtime) | ||
| + | - Submit the job using '' | ||
| + | - Monitor the job using '' | ||
| + | - Review results and resource usage using '' | ||
| - | <code bash> | + | <note tip> |
| - | #!/bin/bash | + | Start with small jobs and modest resource requests. Once your application works correctly, increase CPU, memory, or GPU allocations as needed. |
| - | #SBATCH --partition=main | + | </note> |
| - | #SBATCH -n 4 | + | |
| - | module load openmpi | + | ===== Frequently Asked Questions ===== |
| - | mpirun my_application | + | ==== Which partition should I use? ==== |
| - | </ | + | |
| - | Available OpenMPI versions: | + | * Use '' |
| + | * Use '' | ||
| - | < | + | ==== Do I need the --account option? ==== |
| - | module avail openmpi | + | |
| + | Most VU MIF users do not. | ||
| + | |||
| + | Only users assigned to a project allocation should specify: | ||
| + | |||
| + | < | ||
| + | #SBATCH --account=alloc_xxxx_project | ||
| </ | </ | ||
| - | Load a version: | + | ==== What happens if I omit --time? ==== |
| - | < | + | SLURM applies a default runtime limit of 2 hours. |
| - | module load openmpi/4.0 | + | |
| + | ==== Why is my job pending? ==== | ||
| + | |||
| + | The most common reasons are: | ||
| + | |||
| + | * Insufficient available resources | ||
| + | * Incorrect partition selection | ||
| + | * Resource limits reached | ||
| + | * Invalid allocation account | ||
| + | |||
| + | Check: | ||
| + | |||
| + | < | ||
| + | squeue -j JOBID | ||
| </ | </ | ||
| - | ===== Task Efficiency ===== | + | ==== Where can I find my job output? |
| - | Efficient jobs help everyone use the cluster effectively. | + | By default: |
| - | Good practice: | + | <code text> |
| + | slurm-JOBID.out | ||
| + | </ | ||
| - | * Request only the resources you need | + | in the directory where the job was submitted. |
| - | * Use at least 50% of allocated CPU resources | + | |
| - | * Match application thread count to requested CPUs | + | |
| - | * Set a realistic runtime limit | + | |
| - | Avoid: | + | You can change this using: |
| - | * Requesting excessive CPUs | + | <code shell> |
| - | * Requesting excessive memory | + | #SBATCH --output=myjob.out |
| - | * Reserving GPUs that are not used | + | </ |
| - | * Running large workloads on login nodes | + | |
| ===== Common Problems ===== | ===== Common Problems ===== | ||
| Line 348: | Line 440: | ||
| * Your SLURM account is not yet created | * Your SLURM account is not yet created | ||
| - | * Your allocation does not exist | + | * The allocation |
| * You do not have access to the specified allocation | * You do not have access to the specified allocation | ||
| Line 355: | Line 447: | ||
| Check: | Check: | ||
| - | < | + | < |
| squeue -j JOBID | squeue -j JOBID | ||
| </ | </ | ||
| Line 362: | Line 454: | ||
| * Resources unavailable | * Resources unavailable | ||
| - | * Partition unavailable | + | * Incorrect partition |
| * Resource limits exceeded | * Resource limits exceeded | ||
| Line 369: | Line 461: | ||
| Check the output file: | Check the output file: | ||
| - | < | + | < |
| cat slurm-JOBID.out | cat slurm-JOBID.out | ||
| </ | </ | ||
| - | Also review | + | Also review: |
| - | < | + | < |
| sacct -j JOBID | sacct -j JOBID | ||
| </ | </ | ||
| Line 386: | Line 478: | ||
| * '' | * '' | ||
| * GPU resources are available | * GPU resources are available | ||
| - | + | * Your allocation includes GPU access | |
| - | ===== Useful Commands ===== | + | |
| - | + | ||
| - | ^ Command ^ Purpose ^ | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | '' | + | |
| ===== Next Steps ===== | ===== Next Steps ===== | ||
| - | Now that you know how to submit jobs, learn how resource allocations and project quotas work: | + | Now that you know how to submit |
| - | * [[hpc-new: | + | * [[en:hpc-new: |
| - | * [[hpc-new: | + | * [[en:hpc-new: |
| - | You may also want to revisit: | + | You may also revisit: |
| - | * [[hpc-new: | + | * [[en:hpc-new: |
| - | * [[hpc-new: | + | * [[en:hpc-new: |
| ---- | ---- | ||
en/hpc-new/slurm.txt · Last modified: by eduardas
