====== Batch Job Execution (SLURM) ====== The VU MIF HPC cluster uses SLURM to manage computing resources and execute jobs. If you have used a personal computer before, think of SLURM as a workload manager. Instead of running applications directly, you ask SLURM for resources (CPU cores, memory, GPUs, and runtime), and SLURM finds a suitable compute node to run your job. ===== What Is SLURM? ===== SLURM is responsible for: * Allocating cluster resources * Starting and stopping jobs * Managing queues * Enforcing resource limits * Tracking resource usage Most HPC work starts with creating a job script and submitting it using: sbatch job.sh ===== How Jobs Run ===== A typical HPC workflow looks like this: - Store your files in the shared Lustre file system - Create a SLURM job script - Submit the job using ''sbatch'' - Monitor progress - Review results Unlike a personal computer, computational workloads are normally not executed directly on the login node. Use SLURM to run calculations on compute nodes. ===== Available Partitions ===== The VU MIF HPC cluster currently provides: ^ Partition ^ Purpose ^ Maximum Runtime ^ | ''main'' | General-purpose CPU workloads | 7 days | | ''gpu'' | GPU-accelerated workloads | 48 hours | This guide focuses on the ''main'' and ''gpu'' partitions, which are used by most users. The ''power'' partition is intended for specialized IBM Power9 workloads and is not covered here. Most users should start with the ''main'' partition. Use the ''gpu'' partition only when your application requires NVIDIA GPUs. If no runtime limit is specified, SLURM uses a default runtime limit of 2 hours. ===== Your First Job Script ===== Create a file named ''hello.sh'': nano hello.sh Paste: #!/bin/bash #SBATCH --job-name=hello #SBATCH --partition=main #SBATCH --output=hello.out #SBATCH --time=00:01:00 echo "Hello from HPC!" hostname date Submit the job: sbatch hello.sh Example output: Submitted batch job 123456 ===== Common Resource Requests ===== Many applications require CPU cores, memory, or GPUs. Example: #!/bin/bash #SBATCH --job-name=myjob #SBATCH --partition=main #SBATCH --cpus-per-task=4 #SBATCH --mem=8G #SBATCH --time=01:00:00 python3 script.py ===== Understanding SLURM Directives ===== ^ Directive ^ Description ^ | ''--partition=main'' | Use the CPU partition | | ''--partition=gpu'' | Use the GPU partition | | ''--cpus-per-task=4'' | Request 4 CPU cores | | ''--mem=8G'' | Request 8 GB RAM | | ''--time=01:00:00'' | Request 1 hour runtime | | ''--output=file.out'' | Save output to a file | | ''--job-name=myjob'' | Assign a name to the job | ===== Project Allocations ===== Users working in Waldur-managed projects may need to specify an allocation account: #SBATCH --account=alloc_xxxx_project Example: #!/bin/bash #SBATCH --account=alloc_xxxx_project #SBATCH --partition=main #SBATCH --cpus-per-task=4 #SBATCH --mem=8G #SBATCH --time=01:00:00 python3 script.py VU MIF users using the default allocation do not need to specify the ''--account'' parameter. Only users assigned to a project allocation should use ''--account''. ===== Submitting a Job ===== Submit a job: sbatch job.sh SLURM returns a Job ID: Submitted batch job 123456 The Job ID uniquely identifies your job. ===== Monitoring Jobs ===== Show your running and pending jobs: squeue -u $USER Example: JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 123456 main test user R 00:10 1 compute07 ===== Why Is My Job Pending? ===== A job may remain in ''PENDING'' state when: * Requested resources are currently unavailable * Higher-priority jobs are running * Resource limits have been reached * An incorrect partition or allocation was specified Show detailed status: squeue -j JOBID ===== View Completed Jobs ===== Display information about completed jobs: sacct -j JOBID More detailed output: sacct -j JOBID --format=JobID,State,Elapsed,AllocCPUS Example: JobID State Elapsed AllocCPUS ------------ ---------- ------------ ---------- 123456 COMPLETED 00:01:05 4 ===== Monitor Resource Usage ===== While a job is running: sstat -j JOBID.batch --format=JobID,AveCPU,AveRSS,MaxRSS Example: JobID AveCPU AveRSS MaxRSS ------------ ---------- ---------- ---------- 123456.batch 00:00:01 500M 620M Where: ^ Field ^ Description ^ | ''AveCPU'' | Average CPU time consumed by job tasks | | ''AveRSS'' | Average memory usage | | ''MaxRSS'' | Maximum memory usage | ===== Understanding CPU Usage ===== After a job finishes, you can compare elapsed runtime with total CPU usage: sacct -j JOBID --format=JobID,AllocCPUS,Elapsed,CPUTime Example: JobID AllocCPUS Elapsed CPUTime ------------ ---------- ---------- ---------- 123456 4 00:00:10 00:00:38 ^ Field ^ Description ^ | ''AllocCPUS'' | Number of allocated CPU cores | | ''Elapsed'' | Real (wall-clock) runtime | | ''CPUTime'' | Total CPU time used by all allocated CPU cores | ==== Example: Single-Threaded Application ==== AllocCPUS = 4 Elapsed = 00:00:10 CPUTime = 00:00:10 The application only used approximately one CPU core, even though four cores were allocated. ==== Example: Multi-Threaded Application ==== AllocCPUS = 4 Elapsed = 00:00:10 CPUTime = 00:00:38 The application actively used multiple CPU cores. A fully utilized four-core job running for ten seconds would produce approximately: AllocCPUS = 4 Elapsed = 00:00:10 CPUTime = 00:00:40 A quick way to verify that your application is using multiple CPU cores is to compare ''CPUTime'' with ''Elapsed''. If ''CPUTime'' is several times larger than ''Elapsed'', the workload is utilizing multiple CPU cores concurrently. ===== Cancelling Jobs ===== Cancel a running or pending job: scancel JOBID Example: scancel 123456 ===== Interactive Jobs ===== Sometimes it is useful to work interactively instead of submitting a batch script. Start an interactive shell: srun --pty $SHELL When finished: exit ===== GPU Jobs ===== To reserve GPU resources: #!/bin/bash #SBATCH --partition=gpu #SBATCH --gres=gpu:1 #SBATCH --cpus-per-task=4 #SBATCH --mem=16G #SBATCH --time=01:00:00 nvidia-smi The parameter ''--gres=gpu:1'' requests one GPU. The command: nvidia-smi displays information about the GPU allocated to the job. Users working within a project allocation may also need: #SBATCH --account=alloc_xxxx_project ===== Useful Commands ===== ^ Command ^ Purpose ^ | ''sbatch job.sh'' | Submit a batch job | | ''squeue -u $USER'' | View your jobs | | ''sacct -j JOBID'' | View completed jobs | | ''sstat -j JOBID.batch'' | Monitor running jobs | | ''scancel JOBID'' | Cancel a job | | ''sinfo'' | Show cluster partitions | ===== SLURM Workflow Summary ===== The typical workflow for running a job is: - Create a job script - Request the required resources (CPU, memory, GPU, runtime) - Submit the job using ''sbatch'' - Monitor the job using ''squeue'' - Review results and resource usage using ''sacct'' or ''sstat'' Start with small jobs and modest resource requests. Once your application works correctly, increase CPU, memory, or GPU allocations as needed. ===== Frequently Asked Questions ===== ==== Which partition should I use? ==== * Use ''main'' for most workloads. * Use ''gpu'' only when your application requires NVIDIA GPUs. ==== Do I need the --account option? ==== Most VU MIF users do not. Only users assigned to a project allocation should specify: #SBATCH --account=alloc_xxxx_project ==== What happens if I omit --time? ==== SLURM applies a default runtime limit of 2 hours. ==== Why is my job pending? ==== The most common reasons are: * Insufficient available resources * Incorrect partition selection * Resource limits reached * Invalid allocation account Check: squeue -j JOBID ==== Where can I find my job output? ==== By default: slurm-JOBID.out in the directory where the job was submitted. You can change this using: #SBATCH --output=myjob.out ===== Common Problems ===== ==== Invalid account or account/partition combination specified ==== Possible causes: * Your SLURM account is not yet created * The allocation name does not exist * You do not have access to the specified allocation ==== Job Remains Pending ==== Check: squeue -j JOBID Possible causes: * Resources unavailable * Incorrect partition * Resource limits exceeded ==== Job Exits Immediately ==== Check the output file: cat slurm-JOBID.out Also review: sacct -j JOBID ==== GPU Not Detected ==== Verify: * ''--partition=gpu'' is used * ''--gres=gpu:N'' is specified * GPU resources are available * Your allocation includes GPU access ===== Next Steps ===== Now that you know how to submit and monitor jobs, learn about resource quotas, project allocations, and Waldur-managed projects: * [[en:hpc-new:resources|Resource Allocations and Project Requests]] * [[en:hpc-new:waldur|Waldur User Guide]] You may also revisit: * [[en:hpc-new:getting_started|First Successful Job]] * [[en:hpc-new:lustre|Shared File System (Lustre)]] ---- This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, software stack, and operational policies.