IT wiki

VU MIF STSC

User Tools

Site Tools


en:hpc-new:getting_started

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
en:hpc-new:getting_started [2026/11/19 14:02] – created eduardasen:hpc-new:getting_started [2026/11/19 14:21] (current) – eduardas
Line 4: Line 4:
  
   - Connect to the HPC system   - Connect to the HPC system
 +  - Understand the difference between login and compute nodes
   - Create and submit your first job   - Create and submit your first job
   - Monitor job execution   - Monitor job execution
Line 9: Line 10:
   - Request CPU and memory resources   - Request CPU and memory resources
   - Learn how project allocations work   - Learn how project allocations work
-  - Run a Python application+  - Run a multi-core Python application 
 +  - Verify CPU utilization
   - Submit a GPU job   - Submit a GPU job
  
Line 32: Line 34:
  
 <code bash> <code bash>
-ssh username@login.hpc.mif.vu.lt+ssh username@hpc.mif.vu.lt
 </code> </code>
  
Line 45: Line 47:
  
 You should see the hostname of the login node and your home directory path. You should see the hostname of the login node and your home directory path.
 +
 +<note important>
 +Your SLURM account is created automatically after your first successful login. This process may take up to 5 minutes.
 +
 +Until the account is created, job submission may fail with account-related errors.
 +</note>
  
 For detailed instructions see: For detailed instructions see:
Line 50: Line 58:
   * [[hpc:connection|Connection]]   * [[hpc:connection|Connection]]
  
-===== Step 2: Create a Working Directory =====+===== Step 2: Understanding Login Nodes and Compute Nodes ===== 
 + 
 +Unlike a personal computer, computational workloads are not normally executed directly from the login node. Instead, jobs are submitted to SLURM, which schedules them on available compute nodes. 
 + 
 +The login node is intended for: 
 + 
 +  * Editing files 
 +  * Compiling code 
 +  * Managing data 
 +  * Preparing and submitting jobs 
 + 
 +The compute nodes are used for: 
 + 
 +  * CPU-intensive calculations 
 +  * Simulations 
 +  * Data analysis 
 +  * Machine learning and AI workloads 
 + 
 +<note important> 
 +Do not run computationally intensive applications directly on the login node. Submit them through SLURM so they can be executed on the appropriate compute resources. 
 +</note> 
 + 
 +===== Step 3: Create a Working Directory =====
  
 Create a directory for this tutorial: Create a directory for this tutorial:
  
 <code bash> <code bash>
-mkdir getting-started +mkdir getting_started 
-cd getting-started+cd getting_started
 </code> </code>
  
Line 65: Line 95:
 </code> </code>
  
-===== Step 3: Create Your First Job Script =====+===== Step 4: Create Your First Job Script =====
  
 Create a new file: Create a new file:
Line 89: Line 119:
 Save the file and exit the editor. Save the file and exit the editor.
  
-===== Step 4: Submit the Job =====+===== Step 5: Submit the Job =====
  
 Submit the script to the scheduler: Submit the script to the scheduler:
Line 100: Line 130:
  
 <code> <code>
-Submitted batch job 12345+Submitted batch job 123456
 </code> </code>
  
-The number is your Job ID.+The returned number is your Job ID.
  
-===== Step 5: Monitor the Job =====+===== Step 6: Monitor the Job =====
  
 Check the status of your jobs: Check the status of your jobs:
Line 115: Line 145:
 When the job disappears from the queue, it has finished. When the job disappears from the queue, it has finished.
  
-===== Step 6: View the Results =====+===== Step 7: View the Results =====
  
 Display the output file: Display the output file:
Line 133: Line 163:
 Congratulations! You have successfully executed your first HPC job. Congratulations! You have successfully executed your first HPC job.
  
-===== Step 7: Request Resources for a Job =====+===== Step 8: Request Resources for a Job =====
  
 Real HPC workloads usually require specific CPU, memory, and runtime allocations. These resources are requested using SLURM directives. Real HPC workloads usually require specific CPU, memory, and runtime allocations. These resources are requested using SLURM directives.
Line 154: Line 184:
 #SBATCH --output=resources.out #SBATCH --output=resources.out
  
-echo "Running on:" +echo "Job ID:" 
-hostname+echo $SLURM_JOB_ID
  
 echo "" echo ""
-echo "Allocated CPUs:" +echo "Partition:" 
-nproc+echo $SLURM_JOB_PARTITION
  
 echo "" echo ""
-echo "Available Memory:" +echo "Allocated CPUs:" 
-free -h+echo $SLURM_CPUS_PER_TASK
  
 echo "" echo ""
Line 184: Line 214:
 | ''--time=00:05:00'' | Request up to 5 minutes of runtime | | ''--time=00:05:00'' | Request up to 5 minutes of runtime |
 | ''--output=resources.out'' | Save output to a file | | ''--output=resources.out'' | Save output to a file |
 +
 +<note tip>
 +Most CPU workloads should use the ''main'' partition.
 +
 +GPU workloads should use the ''gpu'' partition.
 +</note>
  
 <note tip> <note tip>
Line 201: Line 237:
 </code> </code>
  
-You should see information about the compute node and the allocated resources. +===== Step 9: Using Project Allocations =====
- +
-===== Step 8: Using Project Allocations =====+
  
 Some users are assigned project allocations that provide additional CPU, GPU, memory, or storage resources. Some users are assigned project allocations that provide additional CPU, GPU, memory, or storage resources.
Line 226: Line 260:
 VU MIF users using the default allocation do NOT need to specify the ''--account'' parameter. VU MIF users using the default allocation do NOT need to specify the ''--account'' parameter.
  
-Only users associated with a project allocation should add:+Only users associated with a project allocation should use the ''--account'' directive. 
 +</note> 
 + 
 +Example:
  
 <code bash> <code bash>
 #SBATCH --account=alloc_xxxx_project #SBATCH --account=alloc_xxxx_project
 </code> </code>
-</note> 
  
-===== Step 9: Run a Python Job =====+===== Step 10: Run a Multi-Core Python Job =====
  
-Python is one of the most commonly used programming languages on HPC systems. This example demonstrates how to execute a Python application through SLURM.+Python is one of the most commonly used programming languages on HPC systems. This example performs a matrix multiplication, a common operation in scientific computing, machine learning, and data analysis.
  
 ==== Create a Python Script ==== ==== Create a Python Script ====
  
-Create a file named ''hello.py'':+Create a file named ''matrix.py'':
  
 <code bash> <code bash>
-nano hello.py+nano matrix.py
 </code> </code>
  
Line 248: Line 284:
  
 <code python> <code python>
-import platform +import os 
-from datetime import datetime+import numpy as np 
 +from time import perf_counter
  
-print("Hello from Python!") +cpus = os.environ.get("SLURM_CPUS_PER_TASK", "unknown")
-print("Hostname:", platform.node()) +
-print("Current time:", datetime.now()) +
-</code>+
  
-Save the file.+print(f"CPU cores allocated by SLURM: {cpus}") 
 + 
 +size = 2000 
 + 
 +print(f"Creating two {size}x{size} matrices...") 
 + 
 +a = np.random.rand(size, size) 
 +b = np.random.rand(size, size) 
 + 
 +start = perf_counter() 
 +c = a @ b 
 +end = perf_counter() 
 + 
 +print("Matrix multiplication completed") 
 +print(f"Result shape: {c.shape}") 
 +print(f"Execution time: {end - start:.2f} seconds") 
 +</code>
  
 ==== Create a SLURM Job Script ==== ==== Create a SLURM Job Script ====
Line 272: Line 322:
 #SBATCH --job-name=python-demo #SBATCH --job-name=python-demo
 #SBATCH --partition=main #SBATCH --partition=main
-#SBATCH --cpus-per-task=1 +#SBATCH --cpus-per-task=4 
-#SBATCH --mem=1G+#SBATCH --mem=2G
 #SBATCH --time=00:05:00 #SBATCH --time=00:05:00
 #SBATCH --output=python-demo.out #SBATCH --output=python-demo.out
  
-python3 hello.py+export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK 
 +export OPENBLAS_NUM_THREADS=$SLURM_CPUS_PER_TASK 
 +export MKL_NUM_THREADS=$SLURM_CPUS_PER_TASK 
 + 
 +python3 matrix.py
 </code> </code>
  
Line 297: Line 351:
  
 <code> <code>
-Hello from Python! +CPU cores allocated by SLURM: 4 
-Hostname: compute01 +Creating two 2000x2000 matrices... 
-Current time: 2026-11-19 15:45:12+Matrix multiplication completed 
 +Result shape: (2000, 2000) 
 +Execution time: 1.82 seconds
 </code> </code>
- 
-Congratulations! You have successfully executed a Python application on the HPC cluster. 
  
 ==== Optional: Load a Python Module ==== ==== Optional: Load a Python Module ====
Line 312: Line 366:
 </code> </code>
  
-Load a version:+Load one of the available versions shown by the system:
  
 <code bash> <code bash>
-module load Python/3.x.y+module load Python/<version>
 </code> </code>
  
Line 324: Line 378:
 </code> </code>
  
-===== Step 10: Submit a GPU Job =====+===== Step 11: Verify CPU Usage ===== 
 + 
 +While a job is running, you can monitor resource utilization with: 
 + 
 +<code bash> 
 +sstat -j JOBID.batch --format=JobID,AveCPU,AveRSS,MaxRSS 
 +</code> 
 + 
 +Example: 
 + 
 +<code> 
 +       JobID     AveCPU     AveRSS     MaxRSS 
 +------------ ---------- ---------- ---------- 
 +123456.batch   00:00:01      500M      620M 
 +</code> 
 + 
 +Where: 
 + 
 +^ Field ^ Description ^ 
 +| ''AveCPU'' | Average CPU time consumed by job tasks | 
 +| ''AveRSS'' | Average memory usage | 
 +| ''MaxRSS'' | Maximum memory usage | 
 + 
 +You can identify your Job ID with: 
 + 
 +<code bash> 
 +squeue -u $USER 
 +</code> 
 + 
 +==== Compare Elapsed Time and CPU Time ==== 
 + 
 +After the job finishes: 
 + 
 +<code bash> 
 +sacct -j JOBID --format=JobID,AllocCPUS,Elapsed,CPUTime 
 +</code> 
 + 
 +Example: 
 + 
 +<code> 
 +JobID          AllocCPUS    Elapsed    CPUTime 
 +------------ ---------- ---------- ---------- 
 +123456                4   00:00:02   00:00:07 
 +</code> 
 + 
 +^ Field ^ Description ^ 
 +| ''AllocCPUS'' | Number of allocated CPU cores | 
 +| ''Elapsed'' | Wall-clock runtime (real time) | 
 +| ''CPUTime'' | Total CPU time used by all allocated CPU cores | 
 + 
 +==== How to Interpret the Results ==== 
 + 
 +Single-threaded application: 
 + 
 +<code> 
 +AllocCPUS = 4 
 +Elapsed   = 00:00:10 
 +CPUTime   = 00:00:10 
 +</code> 
 + 
 +The application used approximately one CPU core, even though four cores were allocated. 
 + 
 +Multi-threaded application: 
 + 
 +<code> 
 +AllocCPUS = 4 
 +Elapsed   = 00:00:10 
 +CPUTime   = 00:00:38 
 +</code> 
 + 
 +The application actively used multiple CPU cores. 
 + 
 +In an ideal case, a fully utilized four-core job running for ten seconds would show: 
 + 
 +<code> 
 +Elapsed   = 00:00:10 
 +CPUTime   = 00:00:40 
 +</code> 
 + 
 +<note tip> 
 +A quick way to verify that your application is using multiple CPU cores is to compare ''CPUTime'' with ''Elapsed''. If ''CPUTime'' is several times larger than ''Elapsed'', the workload is utilizing multiple CPU cores concurrently. 
 +</note> 
 + 
 +<note tip> 
 +Requesting multiple CPU cores with: 
 + 
 +#SBATCH --cpus-per-task=4 
 + 
 +does not automatically make an application run in parallel. The application must also support multithreading and be configured to use the allocated cores. 
 +</note> 
 + 
 +===== Step 12: Submit a GPU Job =====
  
 To run a job on a GPU node: To run a job on a GPU node:
Line 338: Line 483:
 #SBATCH --job-name=gpu-demo #SBATCH --job-name=gpu-demo
 #SBATCH --partition=gpu #SBATCH --partition=gpu
-#SBATCH --gpus=1+#SBATCH --gres=gpu:1
 #SBATCH --cpus-per-task=4 #SBATCH --cpus-per-task=4
 #SBATCH --mem=16G #SBATCH --mem=16G
Line 352: Line 497:
 </code> </code>
  
-The command ''nvidia-smi'' displays information about the allocated GPU.+The command ''nvidia-smi'' displays information about the GPU allocated to the job. 
 + 
 +Users working within a project allocation may also need: 
 + 
 +<code bash> 
 +#SBATCH --account=alloc_xxxx_project 
 +</code> 
 + 
 +==== Interactive GPU Session ==== 
 + 
 +To start an interactive shell on a GPU node: 
 + 
 +<code bash> 
 +srun -p gpu --gres=gpu:1 --pty $SHELL 
 +</code> 
 + 
 +After the session starts: 
 + 
 +<code bash> 
 +nvidia-smi 
 +</code> 
 + 
 +Interactive sessions are useful for testing software, debugging, and experimenting before submitting batch jobs.
  
 ===== Useful Commands ===== ===== Useful Commands =====
Line 363: Line 530:
 | ''squeue -u $USER'' | View your jobs | | ''squeue -u $USER'' | View your jobs |
 | ''sacct -j JOBID'' | View completed job information | | ''sacct -j JOBID'' | View completed job information |
 +| ''sstat -j JOBID.batch'' | Monitor resource usage of a running job |
 | ''scancel JOBID'' | Cancel a job | | ''scancel JOBID'' | Cancel a job |
 | ''cat file'' | Display file contents | | ''cat file'' | Display file contents |
Line 382: Line 550:
   * Submit jobs   * Submit jobs
   * Monitor job execution   * Monitor job execution
-  * Review job outputs +  * Verify resource utilization
-  * Request CPU and memory resources+
   * Run Python applications   * Run Python applications
   * Use project allocations   * Use project allocations
Line 390: Line 557:
 Continue with: Continue with:
  
-  * [[hpc:connection|Connection]] 
   * [[hpc:lustre|Shared File System (Lustre)]]   * [[hpc:lustre|Shared File System (Lustre)]]
   * [[hpc:slurm|Batch Job Execution (SLURM)]]   * [[hpc:slurm|Batch Job Execution (SLURM)]]
Line 408: Line 574:
  
 You are probably running the command on your local computer instead of the HPC login node. You are probably running the command on your local computer instead of the HPC login node.
- 
-Verify your connection: 
- 
-<code bash> 
-hostname 
-</code> 
  
 ==== Invalid account or account/partition combination specified ==== ==== Invalid account or account/partition combination specified ====
Line 422: Line 582:
   * The specified allocation does not exist.   * The specified allocation does not exist.
   * You do not have permission to use the specified allocation.   * You do not have permission to use the specified allocation.
- 
-Wait a few minutes after your first login or contact the project administrator. 
  
 ==== Job remains in PENDING state ==== ==== Job remains in PENDING state ====
Line 434: Line 592:
 squeue -u $USER squeue -u $USER
 </code> </code>
- 
-More information can be found in: 
- 
-  * [[hpc:slurm|Batch Job Execution (SLURM)]] 
  
 ==== GPU job does not start ==== ==== GPU job does not start ====
Line 444: Line 598:
  
   * The job uses ''--partition=gpu''   * The job uses ''--partition=gpu''
 +  * The job requests GPUs using ''--gres=gpu:N''
   * GPU resources are available   * GPU resources are available
   * Your allocation includes GPU access   * Your allocation includes GPU access
Line 460: Line 615:
  
   - [[hpc:registration|Register for HPC Access]]   - [[hpc:registration|Register for HPC Access]]
-  - [[hpc:getting_started|First Successful Job]] 
   - [[hpc:connection|Connection]]   - [[hpc:connection|Connection]]
 +  - [[hpc:getting_started|First Successful Job]]
   - [[hpc:lustre|Shared File System (Lustre)]]   - [[hpc:lustre|Shared File System (Lustre)]]
   - [[hpc:slurm|Batch Job Execution (SLURM)]]   - [[hpc:slurm|Batch Job Execution (SLURM)]]
en/hpc-new/getting_started.1795096929.txt.gz · Last modified: by eduardas

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki