IT wiki

VU MIF STSC

User Tools

Site Tools


en:hpc-new:slurm

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
en:hpc-new:slurm [2026/11/19 14:43] – created eduardasen:hpc-new:slurm [2026/11/19 14:51] (current) – eduardas
Line 1: Line 1:
-====== Shared File System (Lustre) ======+====== Batch Job Execution (SLURM) ======
  
-The VU MIF HPC cluster uses the Lustre shared file system to store user data.+The VU MIF HPC cluster uses SLURM to manage computing resources and execute jobs.
  
-Unlike a personal computer, HPC jobs may run on different compute nodes. Lustre provides a shared storage space that is accessible from the login node and all compute nodes, allowing your jobs to access the same files regardless of where they are executed.+If you have used a personal computer before, think of SLURM as a workload manager. Instead of running applications directly, you ask SLURM for resources (CPU cores, memory, GPUs, and runtime), and SLURM finds a suitable compute node to run your job.
  
-===== What Is Lustre? =====+===== What Is SLURM? =====
  
-Lustre is a high-performance distributed file system designed for HPC environments.+SLURM is responsible for:
  
-It provides:+  * Allocating cluster resources 
 +  * Starting and stopping jobs 
 +  * Managing queues 
 +  * Enforcing resource limits 
 +  * Tracking resource usage
  
-  * Shared access from all cluster nodes +Most HPC work starts with creating a job script and submitting it using:
-  * High throughput for large datasets +
-  * Storage for applications, scripts, and results +
-  * Collaboration capabilities for research projects+
  
-===== Your Home Directory ===== +<code shell> 
- +sbatch job.sh
-Each HPC user receives a home directory: +
- +
-<code> +
-/scratch/lustre/home/username+
 </code> </code>
  
-where ''username'' is your HPC username.+===== How Jobs Run =====
  
-You can quickly navigate to your home directory:+A typical HPC workflow looks like this:
  
-<code bash> +  - Store your files in the shared Lustre file system 
-cd +  - Create a SLURM job script 
-</code>+  - Submit the job using ''sbatch'' 
 +  - Monitor progress 
 +  - Review results
  
-or+<note tip> 
 +Unlike a personal computer, computational workloads are normally not executed directly on the login node.
  
-<code bash> +Use SLURM to run calculations on compute nodes. 
-cd ~ +</note>
-</code>+
  
-Display your current location:+===== Available Partitions =====
  
-<code bash> +The VU MIF HPC cluster currently provides:
-pwd +
-</code>+
  
-===== Why Use Lustre? =====+^ Partition ^ Purpose ^ Maximum Runtime ^ 
 +| ''main'' | General-purpose CPU workloads | 7 days | 
 +| ''gpu'' | GPU-accelerated workloads | 48 hours |
  
-When a job is submitted through SLURM, it may run on any available compute node.+<note> 
 +This guide focuses on the ''main'' and ''gpu'' partitions, which are used by most users.
  
-Files stored on your local computer are not visible to cluster jobs.+The ''power'' partition is intended for specialized IBM Power9 workloads and is not covered here. 
 +</note>
  
-Files stored in Lustre are available everywhere:+<note tip> 
 +Most users should start with the ''main'' partition.
  
-  * Login nodes +Use the ''gpu'' partition only when your application requires NVIDIA GPUs. 
-  * Compute nodes +</note>
-  * Interactive sessions +
-  * Batch jobs+
  
 <note tip> <note tip>
-Store all job scripts, input data, and output files in Lustre. This ensures they will be accessible wherever your job runs.+If no runtime limit is specified, SLURM uses a default runtime limit of 2 hours.
 </note> </note>
  
-===== Creating a Project Directory =====+===== Your First Job Script =====
  
-A good practice is to create a separate directory for each project.+Create a file named ''hello.sh'':
  
-Example:+<code shell> 
 +nano hello.sh 
 +</code>
  
-<code bash> +Paste: 
-mkdir -p ~/projects/my-first-project + 
-cd ~/projects/my-first-project+<code shell> 
 +#!/bin/bash 
 +#SBATCH --job-name=hello 
 +#SBATCH --partition=main 
 +#SBATCH --output=hello.out 
 +#SBATCH --time=00:01:00 
 + 
 +echo "Hello from HPC!" 
 +hostname 
 +date
 </code> </code>
  
-You can then store:+Submit the job:
  
-  * Input data +<code shell> 
-  * Source code +sbatch hello.sh 
-  * Job scripts +</code>
-  * Results +
-  * Documentation+
  
-in a single location.+Example output:
  
-===== Typical Directory Structure =====+<code text> 
 +Submitted batch job 123456 
 +</code> 
 + 
 +===== Common Resource Requests ===== 
 + 
 +Many applications require CPU cores, memory, or GPUs.
  
 Example: Example:
  
-<code> +<code shell> 
-my-project/ +#!/bin/bash 
-├── data/ +#SBATCH --job-name=myjob 
-├── scripts/ +#SBATCH --partition=main 
-├── jobs/ +#SBATCH --cpus-per-task=4 
-├── results/ +#SBATCH --mem=8G 
-└── logs/+#SBATCH --time=01:00:00 
 + 
 +python3 script.py
 </code> </code>
  
-Example commands:+===== Understanding SLURM Directives =====
  
-<code bash> +^ Directive ^ Description ^ 
-mkdir data scripts jobs results logs+| ''--partition=main'' | Use the CPU partition | 
 +| ''--partition=gpu'' | Use the GPU partition | 
 +| ''--cpus-per-task=4'' | Request 4 CPU cores | 
 +| ''--mem=8G'' | Request 8 GB RAM | 
 +| ''--time=01:00:00'' | Request 1 hour runtime | 
 +| ''--output=file.out'' | Save output to a file | 
 +| ''--job-name=myjob'' | Assign a name to the job | 
 + 
 +===== Project Allocations ===== 
 + 
 +Users working in Waldur-managed projects may need to specify an allocation account: 
 + 
 +<code shell> 
 +#SBATCH --account=alloc_xxxx_project
 </code> </code>
  
-===== Checking Available Files =====+Example:
  
-List files:+<code shell> 
 +#!/bin/bash 
 +#SBATCH --account=alloc_xxxx_project 
 +#SBATCH --partition=main 
 +#SBATCH --cpus-per-task=4 
 +#SBATCH --mem=8G 
 +#SBATCH --time=01:00:00
  
-<code bash> +python3 script.py
-ls+
 </code> </code>
  
-List files with details:+<note important> 
 +VU MIF users using the default allocation do not need to specify the ''--account'' parameter.
  
-<code bash> +Only users assigned to a project allocation should use ''--account''. 
-ls -lh+</note> 
 + 
 +===== Submitting a Job ===== 
 + 
 +Submit a job: 
 + 
 +<code shell> 
 +sbatch job.sh
 </code> </code>
  
-Show hidden files:+SLURM returns a Job ID:
  
-<code bash> +<code text> 
-ls -la+Submitted batch job 123456
 </code> </code>
  
-===== Checking Storage Usage =====+The Job ID uniquely identifies your job.
  
-Display the size of the current directory:+===== Monitoring Jobs =====
  
-<code bash> +Show your running and pending jobs: 
-du -sh .+ 
 +<code shell> 
 +squeue -u $USER
 </code> </code>
  
-Display the size of your home directory:+Example:
  
-<code bash> +<code text> 
-du -sh ~+JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 
 +123456 main test user R 00:10 1 compute07
 </code> </code>
  
-Display usage of subdirectories:+===== Why Is My Job Pending? ===== 
 + 
 +A job may remain in ''PENDING'' state when: 
 + 
 +  * Requested resources are currently unavailable 
 +  * Higher-priority jobs are running 
 +  * Resource limits have been reached 
 +  * An incorrect partition or allocation was specified 
 + 
 +Show detailed status:
  
-<code bash> +<code shell> 
-du -sh *+squeue -j JOBID
 </code> </code>
  
-===== Copying Files =====+===== View Completed Jobs =====
  
-Copy a file:+Display information about completed jobs:
  
-<code bash> +<code shell> 
-cp input.txt backup.txt+sacct -j JOBID
 </code> </code>
  
-Copy a directory:+More detailed output:
  
-<code bash> +<code shell> 
-cp -r project project_backup+sacct -j JOBID --format=JobID,State,Elapsed,AllocCPUS
 </code> </code>
  
-Move a file:+Example:
  
-<code bash> +<code text> 
-mv old.txt new.txt+JobID             State      Elapsed  AllocCPUS 
 +------------ ---------- ------------ ---------- 
 +123456       COMPLETED       00:01:05          4
 </code> </code>
  
-Delete a file:+===== Monitor Resource Usage =====
  
-<code bash> +While a job is running: 
-rm file.txt+ 
 +<code shell> 
 +sstat -j JOBID.batch --format=JobID,AveCPU,AveRSS,MaxRSS
 </code> </code>
  
-Delete a directory:+Example:
  
-<code bash> +<code text> 
-rm -r directory+       JobID     AveCPU     AveRSS     MaxRSS 
 +------------ ---------- ---------- ---------- 
 +123456.batch   00:00:01      500M      620M
 </code> </code>
  
-<note important> +Where:
-Be careful when using ''rm''.+
  
-Deleted files cannot usually be recovered. +^ Field ^ Description ^ 
-</note>+| ''AveCPU'' | Average CPU time consumed by job tasks | 
 +| ''AveRSS'' | Average memory usage | 
 +| ''MaxRSS'' | Maximum memory usage |
  
-===== File Transfers =====+===== Understanding CPU Usage =====
  
-Files can be transferred using:+After a job finishes, you can compare elapsed runtime with total CPU usage:
  
-  * SCP +<code shell> 
-  * SFTP +sacct -j JOBID --format=JobID,AllocCPUS,Elapsed,CPUTime 
-  * WinSCP +</code>
-  * Visual Studio Code Remote SSH+
  
-Upload from your computer:+Example:
  
-<code bash> +<code text> 
-scp file.txt username@hpc.mif.vu.lt:~+JobID          AllocCPUS    Elapsed    CPUTime 
 +------------ ---------- ---------- ---------- 
 +123456                4   00:00:10   00:00:38
 </code> </code>
  
-Download to your computer:+^ Field ^ Description ^ 
 +| ''AllocCPUS'' | Number of allocated CPU cores | 
 +| ''Elapsed'' | Real (wall-clock) runtime | 
 +| ''CPUTime'' | Total CPU time used by all allocated CPU cores |
  
-<code bash> +==== Example: Single-Threaded Application ==== 
-scp username@hpc.mif.vu.lt:~/file.txt .+ 
 +<code text> 
 +AllocCPUS = 4 
 +Elapsed   = 00:00:10 
 +CPUTime   = 00:00:10
 </code> </code>
  
-===== Using Lustre with SLURM =====+The application only used approximately one CPU core, even though four cores were allocated.
  
-A job starts in the directory from which it was submitted.+==== Example: Multi-Threaded Application ==== 
 + 
 +<code text> 
 +AllocCPUS = 4 
 +Elapsed   = 00:00:10 
 +CPUTime   = 00:00:38 
 +</code> 
 + 
 +The application actively used multiple CPU cores. 
 + 
 +A fully utilized four-core job running for ten seconds would produce approximately: 
 + 
 +<code text> 
 +AllocCPUS = 4 
 +Elapsed   = 00:00:10 
 +CPUTime   = 00:00:40 
 +</code> 
 + 
 +<note tip> 
 +A quick way to verify that your application is using multiple CPU cores is to compare ''CPUTime'' with ''Elapsed''. 
 + 
 +If ''CPUTime'' is several times larger than ''Elapsed'', the workload is utilizing multiple CPU cores concurrently. 
 +</note> 
 + 
 +===== Cancelling Jobs ===== 
 + 
 +Cancel a running or pending job: 
 + 
 +<code shell> 
 +scancel JOBID 
 +</code>
  
 Example: Example:
  
-<code bash> +<code shell> 
-pwd +scancel 123456
-sbatch job.sh+
 </code> </code>
  
-If the current directory is:+===== Interactive Jobs =====
  
-<code> +Sometimes it is useful to work interactively instead of submitting a batch script. 
-/scratch/lustre/home/username/my-project+ 
 +Start an interactive shell: 
 + 
 +<code shell> 
 +srun --pty $SHELL
 </code> </code>
  
-the job will start in that location.+When finished:
  
-This is why it is important to submit jobs from your project directory.+<code shell> 
 +exit 
 +</code>
  
-===== Verify Where a Job Is Running =====+===== GPU Jobs =====
  
-Create a simple job:+To reserve GPU resources:
  
-<code bash>+<code shell>
 #!/bin/bash #!/bin/bash
-#SBATCH --partition=main +#SBATCH --partition=gpu 
-#SBATCH --output=where.out+#SBATCH --gres=gpu:1 
 +#SBATCH --cpus-per-task=4 
 +#SBATCH --mem=16G 
 +#SBATCH --time=01:00:00
  
-pwd +nvidia-smi
-hostname+
 </code> </code>
  
-After the job finishes:+The parameter ''--gres=gpu:1'' requests one GPU.
  
-<code bash> +The command: 
-cat where.out+ 
 +<code shell> 
 +nvidia-smi
 </code> </code>
  
-Example output:+displays information about the GPU allocated to the job. 
 + 
 +Users working within a project allocation may also need:
  
-<code> +<code shell> 
-/scratch/lustre/home/username/my-project +#SBATCH --account=alloc_xxxx_project
-compute12+
 </code> </code>
  
-Notice that the job runs on a compute node but still accesses your Lustre files.+===== Useful Commands =====
  
-===== Best Practices =====+^ Command ^ Purpose ^ 
 +| ''sbatch job.sh'' | Submit a batch job | 
 +| ''squeue -u $USER'' | View your jobs | 
 +| ''sacct -j JOBID'' | View completed jobs | 
 +| ''sstat -j JOBID.batch'' | Monitor running jobs | 
 +| ''scancel JOBID'' | Cancel a job | 
 +| ''sinfo'' | Show cluster partitions |
  
-✔ Organize work into separate project directories+===== SLURM Workflow Summary =====
  
-✔ Keep job scripts with the associated project+The typical workflow for running a job is:
  
-✔ Store important output files in Lustre+  - Create a job script 
 +  - Request the required resources (CPU, memory, GPU, runtime) 
 +  - Submit the job using ''sbatch'' 
 +  - Monitor the job using ''squeue'' 
 +  - Review results and resource usage using ''sacct'' or ''sstat''
  
-✔ Remove temporary files that are no longer needed+<note tip> 
 +Start with small jobs and modest resource requests. Once your application works correctly, increase CPU, memory, or GPU allocations as needed. 
 +</note>
  
-✔ Use meaningful file names+===== Frequently Asked Questions =====
  
-✔ Keep source code under version control (Git)+==== Which partition should I use? ====
  
-===== Common Mistakes =====+  * Use ''main'' for most workloads. 
 +  * Use ''gpu'' only when your application requires NVIDIA GPUs.
  
-==== Storing Files Only on a Personal Computer ====+==== Do I need the --account option? ====
  
-A compute node cannot access files stored on your laptop.+Most VU MIF users do not.
  
-Always upload required files to Lustre before submitting a job.+Only users assigned to a project allocation should specify:
  
-==== Submitting Jobs from the Wrong Directory ====+<code shell> 
 +#SBATCH --account=alloc_xxxx_project 
 +</code>
  
-Jobs start in the directory where they are submitted.+==== What happens if I omit --time? ====
  
-Verify your location before running:+SLURM applies a default runtime limit of 2 hours.
  
-<code bash> +==== Why is my job pending? ==== 
-pwd+ 
 +The most common reasons are: 
 + 
 +  * Insufficient available resources 
 +  * Incorrect partition selection 
 +  * Resource limits reached 
 +  * Invalid allocation account 
 + 
 +Check: 
 + 
 +<code shell> 
 +squeue -j JOBID
 </code> </code>
  
-==== Accumulating Large Temporary Files ====+==== Where can I find my job output? ====
  
-Large intermediate files can quickly consume storage space.+By default:
  
-Periodically review usage:+<code text> 
 +slurm-JOBID.out 
 +</code> 
 + 
 +in the directory where the job was submitted. 
 + 
 +You can change this using:
  
-<code bash> +<code shell> 
-du -sh ~+#SBATCH --output=myjob.out
 </code> </code>
  
-===== Frequently Asked Questions =====+===== Common Problems =====
  
-==== Can all compute nodes see my files? ====+==== Invalid account or account/partition combination specified ====
  
-Yes.+Possible causes:
  
-Files stored in Lustre are available from all login and compute nodes.+  * Your SLURM account is not yet created 
 +  * The allocation name does not exist 
 +  * You do not have access to the specified allocation
  
-==== Can I share files with collaborators? ====+==== Job Remains Pending ====
  
-Yes.+Check:
  
-Project directories and shared storage can be configured when needed.+<code shell> 
 +squeue -j JOBID 
 +</code>
  
-==== What happens if my job runs on another node? ====+Possible causes:
  
-Your files remain accessible because they are stored in the shared Lustre file system.+  * Resources unavailable 
 +  * Incorrect partition 
 +  * Resource limits exceeded
  
-==== Should I keep data after my project finishes? ====+==== Job Exits Immediately ====
  
-Only retain data that is still needed.+Check the output file:
  
-Unused data should be archived or removed to free storage resources.+<code shell> 
 +cat slurm-JOBID.out 
 +</code>
  
-===== Next Step =====+Also review:
  
-Now that you know where HPC data is stored, learn how to submit more advanced jobs and request resources:+<code shell> 
 +sacct -j JOBID 
 +</code>
  
-  * [[hpc-new:slurm|Batch Job Execution (SLURM)]] +==== GPU Not Detected ==== 
-  * [[hpc-new:resources|Resource Allocations and Project Requests]]+ 
 +Verify: 
 + 
 +  * ''--partition=gpu'' is used 
 +  * ''--gres=gpu:N'' is specified 
 +  * GPU resources are available 
 +  * Your allocation includes GPU access 
 + 
 +===== Next Steps ===== 
 + 
 +Now that you know how to submit and monitor jobs, learn about resource quotas, project allocations, and Waldur-managed projects: 
 + 
 +  * [[en:hpc-new:resources|Resource Allocations and Project Requests]] 
 +  * [[en:hpc-new:waldur|Waldur User Guide]] 
 + 
 +You may also revisit: 
 + 
 +  * [[en:hpc-new:getting_started|First Successful Job]] 
 +  * [[en:hpc-new:lustre|Shared File System (Lustre)]]
  
 ---- ----
Line 327: Line 497:
 This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, software stack, and operational policies. This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, software stack, and operational policies.
 </note> </note>
- 
en/hpc-new/slurm.1795099435.txt.gz · Last modified: by eduardas

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki