IT wiki

VU MIF STSC

User Tools

Site Tools


en:hpc-new:slurm

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
en:hpc-new:slurm [2026/11/19 14:46] – eduardasen:hpc-new:slurm [2026/11/19 14:51] (current) – eduardas
Line 3: Line 3:
 The VU MIF HPC cluster uses SLURM to manage computing resources and execute jobs. The VU MIF HPC cluster uses SLURM to manage computing resources and execute jobs.
  
-If you have used a personal computer before, think of SLURM as a workload manager. Instead of running applications directly, you ask SLURM for resources (CPU cores, memory, GPUs, and runtime) and SLURM finds a suitable compute node to run your job.+If you have used a personal computer before, think of SLURM as a workload manager. Instead of running applications directly, you ask SLURM for resources (CPU cores, memory, GPUs, and runtime), and SLURM finds a suitable compute node to run your job.
  
 ===== What Is SLURM? ===== ===== What Is SLURM? =====
Line 17: Line 17:
 Most HPC work starts with creating a job script and submitting it using: Most HPC work starts with creating a job script and submitting it using:
  
-<code bash>+<code shell>
 sbatch job.sh sbatch job.sh
 </code> </code>
Line 25: Line 25:
 A typical HPC workflow looks like this: A typical HPC workflow looks like this:
  
-  - Prepare your files in Lustre storage+  - Store your files in the shared Lustre file system
   - Create a SLURM job script   - Create a SLURM job script
   - Submit the job using ''sbatch''   - Submit the job using ''sbatch''
Line 44: Line 44:
 | ''main'' | General-purpose CPU workloads | 7 days | | ''main'' | General-purpose CPU workloads | 7 days |
 | ''gpu'' | GPU-accelerated workloads | 48 hours | | ''gpu'' | GPU-accelerated workloads | 48 hours |
 +
 +<note>
 +This guide focuses on the ''main'' and ''gpu'' partitions, which are used by most users.
 +
 +The ''power'' partition is intended for specialized IBM Power9 workloads and is not covered here.
 +</note>
  
 <note tip> <note tip>
Line 59: Line 65:
 Create a file named ''hello.sh'': Create a file named ''hello.sh'':
  
-<code bash>+<code shell>
 nano hello.sh nano hello.sh
 </code> </code>
Line 65: Line 71:
 Paste: Paste:
  
-<code bash>+<code shell>
 #!/bin/bash #!/bin/bash
 #SBATCH --job-name=hello #SBATCH --job-name=hello
Line 79: Line 85:
 Submit the job: Submit the job:
  
-<code bash>+<code shell>
 sbatch hello.sh sbatch hello.sh
 </code> </code>
Line 85: Line 91:
 Example output: Example output:
  
-<code>+<code text>
 Submitted batch job 123456 Submitted batch job 123456
 </code> </code>
Line 95: Line 101:
 Example: Example:
  
-<code bash>+<code shell>
 #!/bin/bash #!/bin/bash
 #SBATCH --job-name=myjob #SBATCH --job-name=myjob
Line 121: Line 127:
 Users working in Waldur-managed projects may need to specify an allocation account: Users working in Waldur-managed projects may need to specify an allocation account:
  
-<code bash>+<code shell>
 #SBATCH --account=alloc_xxxx_project #SBATCH --account=alloc_xxxx_project
 </code> </code>
Line 127: Line 133:
 Example: Example:
  
-<code bash>+<code shell>
 #!/bin/bash #!/bin/bash
 #SBATCH --account=alloc_xxxx_project #SBATCH --account=alloc_xxxx_project
Line 148: Line 154:
 Submit a job: Submit a job:
  
-<code bash>+<code shell>
 sbatch job.sh sbatch job.sh
 </code> </code>
Line 154: Line 160:
 SLURM returns a Job ID: SLURM returns a Job ID:
  
-<code>+<code text>
 Submitted batch job 123456 Submitted batch job 123456
 </code> </code>
Line 164: Line 170:
 Show your running and pending jobs: Show your running and pending jobs:
  
-<code bash>+<code shell>
 squeue -u $USER squeue -u $USER
 </code> </code>
Line 170: Line 176:
 Example: Example:
  
-<code>+<code text>
 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
 123456 main test user R 00:10 1 compute07 123456 main test user R 00:10 1 compute07
Line 186: Line 192:
 Show detailed status: Show detailed status:
  
-<code bash>+<code shell>
 squeue -j JOBID squeue -j JOBID
 </code> </code>
Line 194: Line 200:
 Display information about completed jobs: Display information about completed jobs:
  
-<code bash>+<code shell>
 sacct -j JOBID sacct -j JOBID
 </code> </code>
Line 200: Line 206:
 More detailed output: More detailed output:
  
-<code bash>+<code shell>
 sacct -j JOBID --format=JobID,State,Elapsed,AllocCPUS sacct -j JOBID --format=JobID,State,Elapsed,AllocCPUS
 </code> </code>
Line 206: Line 212:
 Example: Example:
  
-<code> +<code text> 
-JobID             State    Elapsed  AllocCPUS +JobID             State      Elapsed  AllocCPUS 
------------- ---------- ---------- ---------- +------------ ---------- ------------ ---------- 
-123456       COMPLETED    00:01:05          4+123456       COMPLETED       00:01:05          4
 </code> </code>
  
Line 216: Line 222:
 While a job is running: While a job is running:
  
-<code bash>+<code shell>
 sstat -j JOBID.batch --format=JobID,AveCPU,AveRSS,MaxRSS sstat -j JOBID.batch --format=JobID,AveCPU,AveRSS,MaxRSS
 </code> </code>
Line 222: Line 228:
 Example: Example:
  
-<code>+<code text>
        JobID     AveCPU     AveRSS     MaxRSS        JobID     AveCPU     AveRSS     MaxRSS
 ------------ ---------- ---------- ---------- ------------ ---------- ---------- ----------
 123456.batch   00:00:01      500M      620M 123456.batch   00:00:01      500M      620M
 </code> </code>
 +
 +Where:
 +
 +^ Field ^ Description ^
 +| ''AveCPU'' | Average CPU time consumed by job tasks |
 +| ''AveRSS'' | Average memory usage |
 +| ''MaxRSS'' | Maximum memory usage |
 +
 +===== Understanding CPU Usage =====
 +
 +After a job finishes, you can compare elapsed runtime with total CPU usage:
 +
 +<code shell>
 +sacct -j JOBID --format=JobID,AllocCPUS,Elapsed,CPUTime
 +</code>
 +
 +Example:
 +
 +<code text>
 +JobID          AllocCPUS    Elapsed    CPUTime
 +------------ ---------- ---------- ----------
 +123456                4   00:00:10   00:00:38
 +</code>
 +
 +^ Field ^ Description ^
 +| ''AllocCPUS'' | Number of allocated CPU cores |
 +| ''Elapsed'' | Real (wall-clock) runtime |
 +| ''CPUTime'' | Total CPU time used by all allocated CPU cores |
 +
 +==== Example: Single-Threaded Application ====
 +
 +<code text>
 +AllocCPUS = 4
 +Elapsed   = 00:00:10
 +CPUTime   = 00:00:10
 +</code>
 +
 +The application only used approximately one CPU core, even though four cores were allocated.
 +
 +==== Example: Multi-Threaded Application ====
 +
 +<code text>
 +AllocCPUS = 4
 +Elapsed   = 00:00:10
 +CPUTime   = 00:00:38
 +</code>
 +
 +The application actively used multiple CPU cores.
 +
 +A fully utilized four-core job running for ten seconds would produce approximately:
 +
 +<code text>
 +AllocCPUS = 4
 +Elapsed   = 00:00:10
 +CPUTime   = 00:00:40
 +</code>
 +
 +<note tip>
 +A quick way to verify that your application is using multiple CPU cores is to compare ''CPUTime'' with ''Elapsed''.
 +
 +If ''CPUTime'' is several times larger than ''Elapsed'', the workload is utilizing multiple CPU cores concurrently.
 +</note>
  
 ===== Cancelling Jobs ===== ===== Cancelling Jobs =====
Line 232: Line 300:
 Cancel a running or pending job: Cancel a running or pending job:
  
-<code bash>+<code shell>
 scancel JOBID scancel JOBID
 </code> </code>
Line 238: Line 306:
 Example: Example:
  
-<code bash>+<code shell>
 scancel 123456 scancel 123456
 </code> </code>
Line 248: Line 316:
 Start an interactive shell: Start an interactive shell:
  
-<code bash>+<code shell>
 srun --pty $SHELL srun --pty $SHELL
 </code> </code>
Line 254: Line 322:
 When finished: When finished:
  
-<code bash>+<code shell>
 exit exit
 </code> </code>
Line 262: Line 330:
 To reserve GPU resources: To reserve GPU resources:
  
-<code bash>+<code shell>
 #!/bin/bash #!/bin/bash
 #SBATCH --partition=gpu #SBATCH --partition=gpu
Line 273: Line 341:
 </code> </code>
  
-The parameter:+The parameter ''--gres=gpu:1'' requests one GPU.
  
-<code> +The command: 
---gres=gpu:1+ 
 +<code shell> 
 +nvidia-smi
 </code> </code>
  
-requests one GPU.+displays information about the GPU allocated to the job.
  
-===== Interactive GPU Session =====+Users working within a project allocation may also need:
  
-Start an interactive GPU session: +<code shell> 
- +#SBATCH --account=alloc_xxxx_project
-<code bash> +
-srun -p gpu --gres=gpu:1 --pty $SHELL+
 </code> </code>
  
-Verify GPU availability:+===== Useful Commands =====
  
-<code bash> +^ Command ^ Purpose ^ 
-nvidia-smi +| ''sbatch job.sh'' | Submit a batch job | 
-</code>+| ''squeue -u $USER'' | View your jobs | 
 +| ''sacct -j JOBID'' | View completed jobs | 
 +| ''sstat -j JOBID.batch'' | Monitor running jobs | 
 +| ''scancel JOBID'' | Cancel a job | 
 +| ''sinfo'' | Show cluster partitions |
  
-===== MPI Jobs =====+===== SLURM Workflow Summary =====
  
-MPI applications can span multiple CPU cores or nodes.+The typical workflow for running a job is:
  
-Example:+  - Create a job script 
 +  - Request the required resources (CPU, memory, GPU, runtime) 
 +  - Submit the job using ''sbatch'' 
 +  - Monitor the job using ''squeue'' 
 +  - Review results and resource usage using ''sacct'' or ''sstat''
  
-<code bash> +<note tip> 
-#!/bin/bash +Start with small jobs and modest resource requests. Once your application works correctly, increase CPU, memory, or GPU allocations as needed. 
-#SBATCH --partition=main +</note>
-#SBATCH -n 4+
  
-module load openmpi+===== Frequently Asked Questions =====
  
-mpirun my_application +==== Which partition should I use? ====
-</code>+
  
-Available OpenMPI versions:+  * Use ''main'' for most workloads. 
 +  * Use ''gpu'' only when your application requires NVIDIA GPUs.
  
-<code bash> +==== Do I need the --account option? ==== 
-module avail openmpi+ 
 +Most VU MIF users do not. 
 + 
 +Only users assigned to a project allocation should specify: 
 + 
 +<code shell> 
 +#SBATCH --account=alloc_xxxx_project
 </code> </code>
  
-Load a version:+==== What happens if I omit --time? ====
  
-<code bash> +SLURM applies a default runtime limit of 2 hours. 
-module load openmpi/4.0+ 
 +==== Why is my job pending? ==== 
 + 
 +The most common reasons are: 
 + 
 +  * Insufficient available resources 
 +  * Incorrect partition selection 
 +  * Resource limits reached 
 +  * Invalid allocation account 
 + 
 +Check: 
 + 
 +<code shell> 
 +squeue -j JOBID
 </code> </code>
  
-===== Task Efficiency =====+==== Where can I find my job output? ====
  
-Efficient jobs help everyone use the cluster effectively.+By default:
  
-Good practice:+<code text> 
 +slurm-JOBID.out 
 +</code>
  
-  * Request only the resources you need +in the directory where the job was submitted.
-  * Use at least 50% of allocated CPU resources +
-  * Match application thread count to requested CPUs +
-  * Set a realistic runtime limit+
  
-Avoid:+You can change this using:
  
-  * Requesting excessive CPUs +<code shell> 
-  * Requesting excessive memory +#SBATCH --output=myjob.out 
-  * Reserving GPUs that are not used +</code>
-  * Running large workloads on login nodes+
  
 ===== Common Problems ===== ===== Common Problems =====
Line 348: Line 440:
  
   * Your SLURM account is not yet created   * Your SLURM account is not yet created
-  * Your allocation does not exist+  * The allocation name does not exist
   * You do not have access to the specified allocation   * You do not have access to the specified allocation
  
Line 355: Line 447:
 Check: Check:
  
-<code bash>+<code shell>
 squeue -j JOBID squeue -j JOBID
 </code> </code>
Line 362: Line 454:
  
   * Resources unavailable   * Resources unavailable
-  * Partition unavailable+  * Incorrect partition
   * Resource limits exceeded   * Resource limits exceeded
  
Line 369: Line 461:
 Check the output file: Check the output file:
  
-<code bash>+<code shell>
 cat slurm-JOBID.out cat slurm-JOBID.out
 </code> </code>
  
-Also review accounting information:+Also review:
  
-<code bash>+<code shell>
 sacct -j JOBID sacct -j JOBID
 </code> </code>
Line 386: Line 478:
   * ''--gres=gpu:N'' is specified   * ''--gres=gpu:N'' is specified
   * GPU resources are available   * GPU resources are available
- +  * Your allocation includes GPU access
-===== Useful Commands ===== +
- +
-^ Command ^ Purpose ^ +
-| ''sbatch job.sh'' | Submit a batch job | +
-| ''squeue -u $USER'' | View your jobs | +
-| ''sacct -j JOBID'' | View completed jobs | +
-| ''sstat -j JOBID.batch'' | Monitor running jobs | +
-| ''scancel JOBID'' | Cancel a job | +
-| ''sinfo'' | Show cluster partitions |+
  
 ===== Next Steps ===== ===== Next Steps =====
  
-Now that you know how to submit jobs, learn how resource allocations and project quotas work:+Now that you know how to submit and monitor jobs, learn about resource quotas, project allocations, and Waldur-managed projects:
  
-  * [[hpc-new:resources|Resource Allocations and Project Requests]] +  * [[en:hpc-new:resources|Resource Allocations and Project Requests]] 
-  * [[hpc-new:waldur|Waldur User Guide]]+  * [[en:hpc-new:waldur|Waldur User Guide]]
  
-You may also want to revisit:+You may also revisit:
  
-  * [[hpc-new:getting_started|First Successful Job]] +  * [[en:hpc-new:getting_started|First Successful Job]] 
-  * [[hpc-new:lustre|Shared File System (Lustre)]]+  * [[en:hpc-new:lustre|Shared File System (Lustre)]]
  
 ---- ----
en/hpc-new/slurm.txt · Last modified: by eduardas

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki