en:hpc-new:slurm
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
| en:hpc-new:slurm [2026/11/19 14:43] – created eduardas | en:hpc-new:slurm [2026/11/19 14:51] (current) – eduardas | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== | + | ====== |
| - | The VU MIF HPC cluster uses the Lustre shared file system | + | The VU MIF HPC cluster uses SLURM to manage computing resources and execute jobs. |
| - | Unlike | + | If you have used a personal computer |
| - | ===== What Is Lustre? ===== | + | ===== What Is SLURM? ===== |
| - | Lustre | + | SLURM is responsible |
| - | It provides: | + | * Allocating cluster resources |
| + | * Starting and stopping jobs | ||
| + | * Managing queues | ||
| + | * Enforcing resource limits | ||
| + | * Tracking resource usage | ||
| - | * Shared access from all cluster nodes | + | Most HPC work starts with creating a job script |
| - | * High throughput for large datasets | + | |
| - | * Storage for applications, | + | |
| - | * Collaboration capabilities for research projects | + | |
| - | ===== Your Home Directory ===== | + | < |
| - | + | sbatch job.sh | |
| - | Each HPC user receives a home directory: | + | |
| - | + | ||
| - | < | + | |
| - | / | + | |
| </ | </ | ||
| - | where '' | + | ===== How Jobs Run ===== |
| - | You can quickly navigate to your home directory: | + | A typical HPC workflow looks like this: |
| - | <code bash> | + | - Store your files in the shared Lustre file system |
| - | cd | + | - Create a SLURM job script |
| - | </ | + | - Submit the job using '' |
| + | - Monitor progress | ||
| + | - Review results | ||
| - | or | + | <note tip> |
| + | Unlike a personal computer, computational workloads are normally not executed directly on the login node. | ||
| - | <code bash> | + | Use SLURM to run calculations on compute nodes. |
| - | cd ~ | + | </note> |
| - | </code> | + | |
| - | Display your current location: | + | ===== Available Partitions ===== |
| - | <code bash> | + | The VU MIF HPC cluster currently provides: |
| - | pwd | + | |
| - | </ | + | |
| - | ===== Why Use Lustre? ===== | + | ^ Partition ^ Purpose ^ Maximum Runtime ^ |
| + | | '' | ||
| + | | '' | ||
| - | When a job is submitted through SLURM, it may run on any available compute node. | + | < |
| + | This guide focuses | ||
| - | Files stored on your local computer are not visible to cluster jobs. | + | The '' |
| + | </ | ||
| - | Files stored in Lustre are available everywhere: | + | <note tip> |
| + | Most users should start with the '' | ||
| - | * Login nodes | + | Use the '' |
| - | * Compute nodes | + | </ |
| - | * Interactive sessions | + | |
| - | * Batch jobs | + | |
| <note tip> | <note tip> | ||
| - | Store all job scripts, input data, and output files in Lustre. This ensures they will be accessible wherever your job runs. | + | If no runtime limit is specified, SLURM uses a default runtime limit of 2 hours. |
| </ | </ | ||
| - | ===== Creating a Project Directory | + | ===== Your First Job Script |
| - | A good practice is to create | + | Create |
| - | Example: | + | <code shell> |
| + | nano hello.sh | ||
| + | </ | ||
| - | < | + | Paste: |
| - | mkdir -p ~/projects/my-first-project | + | |
| - | cd ~/ | + | < |
| + | #!/bin/bash | ||
| + | # | ||
| + | # | ||
| + | #SBATCH --output=hello.out | ||
| + | #SBATCH --time=00: | ||
| + | |||
| + | echo "Hello from HPC!" | ||
| + | hostname | ||
| + | date | ||
| </ | </ | ||
| - | You can then store: | + | Submit the job: |
| - | * Input data | + | <code shell> |
| - | * Source | + | sbatch hello.sh |
| - | * Job scripts | + | </ |
| - | * Results | + | |
| - | * Documentation | + | |
| - | in a single location. | + | Example output: |
| - | ===== Typical Directory Structure | + | <code text> |
| + | Submitted batch job 123456 | ||
| + | </ | ||
| + | |||
| + | ===== Common Resource Requests | ||
| + | |||
| + | Many applications require CPU cores, memory, or GPUs. | ||
| Example: | Example: | ||
| - | < | + | < |
| - | my-project/ | + | #!/bin/bash |
| - | ├── data/ | + | #SBATCH --job-name=myjob |
| - | ├── scripts/ | + | #SBATCH --partition=main |
| - | ├── jobs/ | + | #SBATCH --cpus-per-task=4 |
| - | ├── results/ | + | #SBATCH --mem=8G |
| - | └── logs/ | + | #SBATCH --time=01: |
| + | |||
| + | python3 script.py | ||
| </ | </ | ||
| - | Example commands: | + | ===== Understanding SLURM Directives ===== |
| - | < | + | ^ Directive ^ Description ^ |
| - | mkdir data scripts jobs results logs | + | | '' |
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | |||
| + | ===== Project Allocations ===== | ||
| + | |||
| + | Users working in Waldur-managed projects may need to specify an allocation account: | ||
| + | |||
| + | < | ||
| + | #SBATCH --account=alloc_xxxx_project | ||
| </ | </ | ||
| - | ===== Checking Available Files ===== | + | Example: |
| - | List files: | + | <code shell> |
| + | # | ||
| + | #SBATCH --account=alloc_xxxx_project | ||
| + | #SBATCH --partition=main | ||
| + | #SBATCH --cpus-per-task=4 | ||
| + | #SBATCH --mem=8G | ||
| + | #SBATCH --time=01:00:00 | ||
| - | <code bash> | + | python3 script.py |
| - | ls | + | |
| </ | </ | ||
| - | List files with details: | + | <note important> |
| + | VU MIF users using the default allocation do not need to specify the '' | ||
| - | < | + | Only users assigned to a project allocation should use '' |
| - | ls -lh | + | </ |
| + | |||
| + | ===== Submitting a Job ===== | ||
| + | |||
| + | Submit a job: | ||
| + | |||
| + | < | ||
| + | sbatch job.sh | ||
| </ | </ | ||
| - | Show hidden files: | + | SLURM returns a Job ID: |
| - | < | + | < |
| - | ls -la | + | Submitted batch job 123456 |
| </ | </ | ||
| - | ===== Checking Storage Usage ===== | + | The Job ID uniquely identifies your job. |
| - | Display the size of the current directory: | + | ===== Monitoring Jobs ===== |
| - | < | + | Show your running and pending jobs: |
| - | du -sh . | + | |
| + | < | ||
| + | squeue | ||
| </ | </ | ||
| - | Display the size of your home directory: | + | Example: |
| - | < | + | < |
| - | du -sh ~ | + | JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) |
| + | 123456 main test user R 00:10 1 compute07 | ||
| </ | </ | ||
| - | Display usage of subdirectories: | + | ===== Why Is My Job Pending? ===== |
| + | |||
| + | A job may remain in '' | ||
| + | |||
| + | * Requested resources are currently unavailable | ||
| + | * Higher-priority jobs are running | ||
| + | * Resource limits have been reached | ||
| + | * An incorrect partition or allocation was specified | ||
| + | |||
| + | Show detailed status: | ||
| - | < | + | < |
| - | du -sh * | + | squeue |
| </ | </ | ||
| - | ===== Copying Files ===== | + | ===== View Completed Jobs ===== |
| - | Copy a file: | + | Display information about completed jobs: |
| - | < | + | < |
| - | cp input.txt backup.txt | + | sacct -j JOBID |
| </ | </ | ||
| - | Copy a directory: | + | More detailed output: |
| - | < | + | < |
| - | cp -r project project_backup | + | sacct -j JOBID --format=JobID, |
| </ | </ | ||
| - | Move a file: | + | Example: |
| - | < | + | < |
| - | mv old.txt new.txt | + | JobID |
| + | ------------ ---------- ------------ ---------- | ||
| + | 123456 | ||
| </ | </ | ||
| - | Delete a file: | + | ===== Monitor Resource Usage ===== |
| - | < | + | While a job is running: |
| - | rm file.txt | + | |
| + | < | ||
| + | sstat -j JOBID.batch --format=JobID, | ||
| </ | </ | ||
| - | Delete a directory: | + | Example: |
| - | < | + | < |
| - | rm -r directory | + | JobID |
| + | ------------ ---------- ---------- ---------- | ||
| + | 123456.batch | ||
| </ | </ | ||
| - | <note important> | + | Where: |
| - | Be careful when using '' | + | |
| - | Deleted files cannot usually be recovered. | + | ^ Field ^ Description ^ |
| - | </ | + | | '' |
| + | | '' | ||
| + | | '' | ||
| - | ===== File Transfers | + | ===== Understanding CPU Usage ===== |
| - | Files can be transferred using: | + | After a job finishes, you can compare elapsed runtime with total CPU usage: |
| - | * SCP | + | <code shell> |
| - | * SFTP | + | sacct -j JOBID --format=JobID, |
| - | * WinSCP | + | </ |
| - | * Visual Studio Code Remote SSH | + | |
| - | Upload from your computer: | + | Example: |
| - | < | + | < |
| - | scp file.txt username@hpc.mif.vu.lt:~ | + | JobID AllocCPUS |
| + | ------------ ---------- ---------- ---------- | ||
| + | 123456 | ||
| </ | </ | ||
| - | Download to your computer: | + | ^ Field ^ Description ^ |
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| - | < | + | ==== Example: Single-Threaded Application ==== |
| - | scp username@hpc.mif.vu.lt:~/file.txt . | + | |
| + | < | ||
| + | AllocCPUS = 4 | ||
| + | Elapsed | ||
| + | CPUTime | ||
| </ | </ | ||
| - | ===== Using Lustre with SLURM ===== | + | The application only used approximately one CPU core, even though four cores were allocated. |
| - | A job starts in the directory from which it was submitted. | + | ==== Example: Multi-Threaded Application ==== |
| + | |||
| + | <code text> | ||
| + | AllocCPUS = 4 | ||
| + | Elapsed | ||
| + | CPUTime | ||
| + | </ | ||
| + | |||
| + | The application actively used multiple CPU cores. | ||
| + | |||
| + | A fully utilized four-core | ||
| + | |||
| + | <code text> | ||
| + | AllocCPUS = 4 | ||
| + | Elapsed | ||
| + | CPUTime | ||
| + | </ | ||
| + | |||
| + | <note tip> | ||
| + | A quick way to verify that your application is using multiple CPU cores is to compare '' | ||
| + | |||
| + | If '' | ||
| + | </ | ||
| + | |||
| + | ===== Cancelling Jobs ===== | ||
| + | |||
| + | Cancel a running or pending job: | ||
| + | |||
| + | <code shell> | ||
| + | scancel JOBID | ||
| + | </ | ||
| Example: | Example: | ||
| - | < | + | < |
| - | pwd | + | scancel 123456 |
| - | sbatch job.sh | + | |
| </ | </ | ||
| - | If the current directory is: | + | ===== Interactive Jobs ===== |
| - | < | + | Sometimes it is useful to work interactively instead of submitting a batch script. |
| - | / | + | |
| + | Start an interactive shell: | ||
| + | |||
| + | < | ||
| + | srun --pty $SHELL | ||
| </ | </ | ||
| - | the job will start in that location. | + | When finished: |
| - | This is why it is important to submit jobs from your project directory. | + | <code shell> |
| + | exit | ||
| + | </ | ||
| - | ===== Verify Where a Job Is Running | + | ===== GPU Jobs ===== |
| - | Create a simple job: | + | To reserve GPU resources: |
| - | < | + | < |
| #!/bin/bash | #!/bin/bash | ||
| - | #SBATCH --partition=main | + | #SBATCH --partition=gpu |
| - | #SBATCH --output=where.out | + | #SBATCH --gres=gpu:1 |
| + | #SBATCH --cpus-per-task=4 | ||
| + | #SBATCH --mem=16G | ||
| + | #SBATCH --time=01: | ||
| - | pwd | + | nvidia-smi |
| - | hostname | + | |
| </ | </ | ||
| - | After the job finishes: | + | The parameter '' |
| - | < | + | The command: |
| - | cat where.out | + | |
| + | < | ||
| + | nvidia-smi | ||
| </ | </ | ||
| - | Example output: | + | displays information about the GPU allocated to the job. |
| + | |||
| + | Users working within a project allocation may also need: | ||
| - | < | + | < |
| - | / | + | # |
| - | compute12 | + | |
| </ | </ | ||
| - | Notice that the job runs on a compute node but still accesses your Lustre files. | + | ===== Useful Commands ===== |
| - | ===== Best Practices ===== | + | ^ Command ^ Purpose ^ |
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| - | ✔ Organize work into separate project directories | + | ===== SLURM Workflow Summary ===== |
| - | ✔ Keep job scripts with the associated project | + | The typical workflow for running a job is: |
| - | ✔ Store important output files in Lustre | + | - Create a job script |
| + | - Request the required resources (CPU, memory, GPU, runtime) | ||
| + | - Submit the job using '' | ||
| + | - Monitor the job using '' | ||
| + | - Review results and resource usage using '' | ||
| - | ✔ Remove temporary files that are no longer | + | <note tip> |
| + | Start with small jobs and modest resource requests. Once your application works correctly, increase CPU, memory, or GPU allocations as needed. | ||
| + | </ | ||
| - | ✔ Use meaningful file names | + | ===== Frequently Asked Questions ===== |
| - | ✔ Keep source code under version control (Git) | + | ==== Which partition should I use? ==== |
| - | ===== Common Mistakes ===== | + | * Use '' |
| + | * Use '' | ||
| - | ==== Storing Files Only on a Personal Computer | + | ==== Do I need the --account option? |
| - | A compute node cannot access files stored on your laptop. | + | Most VU MIF users do not. |
| - | Always upload required files to Lustre before submitting | + | Only users assigned |
| - | ==== Submitting Jobs from the Wrong Directory ==== | + | <code shell> |
| + | #SBATCH --account=alloc_xxxx_project | ||
| + | </ | ||
| - | Jobs start in the directory where they are submitted. | + | ==== What happens if I omit --time? ==== |
| - | Verify your location before running: | + | SLURM applies a default runtime limit of 2 hours. |
| - | < | + | ==== Why is my job pending? ==== |
| - | pwd | + | |
| + | The most common reasons are: | ||
| + | |||
| + | * Insufficient available resources | ||
| + | * Incorrect partition selection | ||
| + | * Resource limits reached | ||
| + | * Invalid allocation account | ||
| + | |||
| + | Check: | ||
| + | |||
| + | < | ||
| + | squeue -j JOBID | ||
| </ | </ | ||
| - | ==== Accumulating Large Temporary Files ==== | + | ==== Where can I find my job output? |
| - | Large intermediate files can quickly consume storage space. | + | By default: |
| - | Periodically review usage: | + | <code text> |
| + | slurm-JOBID.out | ||
| + | </ | ||
| + | |||
| + | in the directory where the job was submitted. | ||
| + | |||
| + | You can change this using: | ||
| - | < | + | < |
| - | du -sh ~ | + | # |
| </ | </ | ||
| - | ===== Frequently Asked Questions | + | ===== Common Problems |
| - | ==== Can all compute nodes see my files? | + | ==== Invalid account or account/ |
| - | Yes. | + | Possible causes: |
| - | Files stored in Lustre are available from all login and compute nodes. | + | * Your SLURM account is not yet created |
| + | * The allocation name does not exist | ||
| + | * You do not have access to the specified allocation | ||
| - | ==== Can I share files with collaborators? | + | ==== Job Remains Pending |
| - | Yes. | + | Check: |
| - | Project directories and shared storage can be configured when needed. | + | <code shell> |
| + | squeue -j JOBID | ||
| + | </ | ||
| - | ==== What happens if my job runs on another node? ==== | + | Possible causes: |
| - | Your files remain accessible because they are stored in the shared Lustre file system. | + | * Resources unavailable |
| + | * Incorrect partition | ||
| + | * Resource limits exceeded | ||
| - | ==== Should I keep data after my project finishes? | + | ==== Job Exits Immediately |
| - | Only retain data that is still needed. | + | Check the output file: |
| - | Unused data should be archived or removed to free storage resources. | + | <code shell> |
| + | cat slurm-JOBID.out | ||
| + | </ | ||
| - | ===== Next Step ===== | + | Also review: |
| - | Now that you know where HPC data is stored, learn how to submit more advanced jobs and request resources: | + | <code shell> |
| + | sacct -j JOBID | ||
| + | </ | ||
| - | | + | ==== GPU Not Detected ==== |
| - | * [[hpc-new:resources|Resource Allocations and Project Requests]] | + | |
| + | Verify: | ||
| + | |||
| + | * '' | ||
| + | * '' | ||
| + | * GPU resources are available | ||
| + | * Your allocation includes GPU access | ||
| + | |||
| + | ===== Next Steps ===== | ||
| + | |||
| + | Now that you know how to submit and monitor jobs, learn about resource quotas, project allocations, | ||
| + | |||
| + | | ||
| + | * [[en: | ||
| + | |||
| + | You may also revisit: | ||
| + | |||
| + | * [[en: | ||
| + | * [[en:hpc-new:lustre|Shared File System (Lustre)]] | ||
| ---- | ---- | ||
| Line 327: | Line 497: | ||
| This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, | This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, | ||
| </ | </ | ||
| - | |||
en/hpc-new/slurm.1795099435.txt.gz · Last modified: by eduardas
