en:hpc-new:getting_started
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
| en:hpc-new:getting_started [2026/11/19 14:02] – created eduardas | en:hpc-new:getting_started [2026/11/19 14:21] (current) – eduardas | ||
|---|---|---|---|
| Line 4: | Line 4: | ||
| - Connect to the HPC system | - Connect to the HPC system | ||
| + | - Understand the difference between login and compute nodes | ||
| - Create and submit your first job | - Create and submit your first job | ||
| - Monitor job execution | - Monitor job execution | ||
| Line 9: | Line 10: | ||
| - Request CPU and memory resources | - Request CPU and memory resources | ||
| - Learn how project allocations work | - Learn how project allocations work | ||
| - | - Run a Python application | + | - Run a multi-core |
| + | - Verify CPU utilization | ||
| - Submit a GPU job | - Submit a GPU job | ||
| Line 32: | Line 34: | ||
| <code bash> | <code bash> | ||
| - | ssh username@login.hpc.mif.vu.lt | + | ssh username@hpc.mif.vu.lt |
| </ | </ | ||
| Line 45: | Line 47: | ||
| You should see the hostname of the login node and your home directory path. | You should see the hostname of the login node and your home directory path. | ||
| + | |||
| + | <note important> | ||
| + | Your SLURM account is created automatically after your first successful login. This process may take up to 5 minutes. | ||
| + | |||
| + | Until the account is created, job submission may fail with account-related errors. | ||
| + | </ | ||
| For detailed instructions see: | For detailed instructions see: | ||
| Line 50: | Line 58: | ||
| * [[hpc: | * [[hpc: | ||
| - | ===== Step 2: Create a Working Directory ===== | + | ===== Step 2: Understanding Login Nodes and Compute Nodes ===== |
| + | |||
| + | Unlike a personal computer, computational workloads are not normally executed directly from the login node. Instead, jobs are submitted to SLURM, which schedules them on available compute nodes. | ||
| + | |||
| + | The login node is intended for: | ||
| + | |||
| + | * Editing files | ||
| + | * Compiling code | ||
| + | * Managing data | ||
| + | * Preparing and submitting jobs | ||
| + | |||
| + | The compute nodes are used for: | ||
| + | |||
| + | * CPU-intensive calculations | ||
| + | * Simulations | ||
| + | * Data analysis | ||
| + | * Machine learning and AI workloads | ||
| + | |||
| + | <note important> | ||
| + | Do not run computationally intensive applications directly on the login node. Submit them through SLURM so they can be executed on the appropriate compute resources. | ||
| + | </ | ||
| + | |||
| + | ===== Step 3: Create a Working Directory ===== | ||
| Create a directory for this tutorial: | Create a directory for this tutorial: | ||
| <code bash> | <code bash> | ||
| - | mkdir getting-started | + | mkdir getting_started |
| - | cd getting-started | + | cd getting_started |
| </ | </ | ||
| Line 65: | Line 95: | ||
| </ | </ | ||
| - | ===== Step 3: Create Your First Job Script ===== | + | ===== Step 4: Create Your First Job Script ===== |
| Create a new file: | Create a new file: | ||
| Line 89: | Line 119: | ||
| Save the file and exit the editor. | Save the file and exit the editor. | ||
| - | ===== Step 4: Submit the Job ===== | + | ===== Step 5: Submit the Job ===== |
| Submit the script to the scheduler: | Submit the script to the scheduler: | ||
| Line 100: | Line 130: | ||
| < | < | ||
| - | Submitted batch job 12345 | + | Submitted batch job 123456 |
| </ | </ | ||
| - | The number is your Job ID. | + | The returned |
| - | ===== Step 5: Monitor the Job ===== | + | ===== Step 6: Monitor the Job ===== |
| Check the status of your jobs: | Check the status of your jobs: | ||
| Line 115: | Line 145: | ||
| When the job disappears from the queue, it has finished. | When the job disappears from the queue, it has finished. | ||
| - | ===== Step 6: View the Results ===== | + | ===== Step 7: View the Results ===== |
| Display the output file: | Display the output file: | ||
| Line 133: | Line 163: | ||
| Congratulations! You have successfully executed your first HPC job. | Congratulations! You have successfully executed your first HPC job. | ||
| - | ===== Step 7: Request Resources for a Job ===== | + | ===== Step 8: Request Resources for a Job ===== |
| Real HPC workloads usually require specific CPU, memory, and runtime allocations. These resources are requested using SLURM directives. | Real HPC workloads usually require specific CPU, memory, and runtime allocations. These resources are requested using SLURM directives. | ||
| Line 154: | Line 184: | ||
| #SBATCH --output=resources.out | #SBATCH --output=resources.out | ||
| - | echo "Running on:" | + | echo "Job ID:" |
| - | hostname | + | echo $SLURM_JOB_ID |
| echo "" | echo "" | ||
| - | echo "Allocated CPUs:" | + | echo "Partition:" |
| - | nproc | + | echo $SLURM_JOB_PARTITION |
| echo "" | echo "" | ||
| - | echo "Available Memory:" | + | echo "Allocated CPUs:" |
| - | free -h | + | echo $SLURM_CPUS_PER_TASK |
| echo "" | echo "" | ||
| Line 184: | Line 214: | ||
| | '' | | '' | ||
| | '' | | '' | ||
| + | |||
| + | <note tip> | ||
| + | Most CPU workloads should use the '' | ||
| + | |||
| + | GPU workloads should use the '' | ||
| + | </ | ||
| <note tip> | <note tip> | ||
| Line 201: | Line 237: | ||
| </ | </ | ||
| - | You should see information about the compute node and the allocated resources. | + | ===== Step 9: Using Project Allocations ===== |
| - | + | ||
| - | ===== Step 8: Using Project Allocations ===== | + | |
| Some users are assigned project allocations that provide additional CPU, GPU, memory, or storage resources. | Some users are assigned project allocations that provide additional CPU, GPU, memory, or storage resources. | ||
| Line 226: | Line 260: | ||
| VU MIF users using the default allocation do NOT need to specify the '' | VU MIF users using the default allocation do NOT need to specify the '' | ||
| - | Only users associated with a project allocation should | + | Only users associated with a project allocation should |
| + | </ | ||
| + | |||
| + | Example: | ||
| <code bash> | <code bash> | ||
| #SBATCH --account=alloc_xxxx_project | #SBATCH --account=alloc_xxxx_project | ||
| </ | </ | ||
| - | </ | ||
| - | ===== Step 9: Run a Python Job ===== | + | ===== Step 10: Run a Multi-Core |
| - | Python is one of the most commonly used programming languages on HPC systems. This example | + | Python is one of the most commonly used programming languages on HPC systems. This example |
| ==== Create a Python Script ==== | ==== Create a Python Script ==== | ||
| - | Create a file named '' | + | Create a file named '' |
| <code bash> | <code bash> | ||
| - | nano hello.py | + | nano matrix.py |
| </ | </ | ||
| Line 248: | Line 284: | ||
| <code python> | <code python> | ||
| - | import | + | import |
| - | from datetime | + | import numpy as np |
| + | from time import | ||
| - | print("Hello from Python!" | + | cpus = os.environ.get("SLURM_CPUS_PER_TASK", "unknown") |
| - | print(" | + | |
| - | print("Current time:", datetime.now()) | + | |
| - | </ | + | |
| - | Save the file. | + | print(f" |
| + | |||
| + | size = 2000 | ||
| + | |||
| + | print(f" | ||
| + | |||
| + | a = np.random.rand(size, | ||
| + | b = np.random.rand(size, | ||
| + | |||
| + | start = perf_counter() | ||
| + | c = a @ b | ||
| + | end = perf_counter() | ||
| + | |||
| + | print(" | ||
| + | print(f" | ||
| + | print(f" | ||
| + | </ | ||
| ==== Create a SLURM Job Script ==== | ==== Create a SLURM Job Script ==== | ||
| Line 272: | Line 322: | ||
| #SBATCH --job-name=python-demo | #SBATCH --job-name=python-demo | ||
| #SBATCH --partition=main | #SBATCH --partition=main | ||
| - | #SBATCH --cpus-per-task=1 | + | #SBATCH --cpus-per-task=4 |
| - | #SBATCH --mem=1G | + | #SBATCH --mem=2G |
| #SBATCH --time=00: | #SBATCH --time=00: | ||
| #SBATCH --output=python-demo.out | #SBATCH --output=python-demo.out | ||
| - | python3 | + | export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK |
| + | export OPENBLAS_NUM_THREADS=$SLURM_CPUS_PER_TASK | ||
| + | export MKL_NUM_THREADS=$SLURM_CPUS_PER_TASK | ||
| + | |||
| + | python3 | ||
| </ | </ | ||
| Line 297: | Line 351: | ||
| < | < | ||
| - | Hello from Python! | + | CPU cores allocated by SLURM: 4 |
| - | Hostname: compute01 | + | Creating two 2000x2000 matrices... |
| - | Current | + | Matrix multiplication completed |
| + | Result shape: (2000, 2000) | ||
| + | Execution | ||
| </ | </ | ||
| - | |||
| - | Congratulations! You have successfully executed a Python application on the HPC cluster. | ||
| ==== Optional: Load a Python Module ==== | ==== Optional: Load a Python Module ==== | ||
| Line 312: | Line 366: | ||
| </ | </ | ||
| - | Load a version: | + | Load one of the available versions shown by the system: |
| <code bash> | <code bash> | ||
| - | module load Python/3.x.y | + | module load Python/< |
| </ | </ | ||
| Line 324: | Line 378: | ||
| </ | </ | ||
| - | ===== Step 10: Submit a GPU Job ===== | + | ===== Step 11: Verify CPU Usage ===== |
| + | |||
| + | While a job is running, you can monitor resource utilization with: | ||
| + | |||
| + | <code bash> | ||
| + | sstat -j JOBID.batch --format=JobID, | ||
| + | </ | ||
| + | |||
| + | Example: | ||
| + | |||
| + | < | ||
| + | | ||
| + | ------------ ---------- ---------- ---------- | ||
| + | 123456.batch | ||
| + | </ | ||
| + | |||
| + | Where: | ||
| + | |||
| + | ^ Field ^ Description ^ | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | |||
| + | You can identify your Job ID with: | ||
| + | |||
| + | <code bash> | ||
| + | squeue -u $USER | ||
| + | </ | ||
| + | |||
| + | ==== Compare Elapsed Time and CPU Time ==== | ||
| + | |||
| + | After the job finishes: | ||
| + | |||
| + | <code bash> | ||
| + | sacct -j JOBID --format=JobID, | ||
| + | </ | ||
| + | |||
| + | Example: | ||
| + | |||
| + | < | ||
| + | JobID AllocCPUS | ||
| + | ------------ ---------- ---------- ---------- | ||
| + | 123456 | ||
| + | </ | ||
| + | |||
| + | ^ Field ^ Description ^ | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | |||
| + | ==== How to Interpret the Results ==== | ||
| + | |||
| + | Single-threaded application: | ||
| + | |||
| + | < | ||
| + | AllocCPUS = 4 | ||
| + | Elapsed | ||
| + | CPUTime | ||
| + | </ | ||
| + | |||
| + | The application used approximately one CPU core, even though four cores were allocated. | ||
| + | |||
| + | Multi-threaded application: | ||
| + | |||
| + | < | ||
| + | AllocCPUS = 4 | ||
| + | Elapsed | ||
| + | CPUTime | ||
| + | </ | ||
| + | |||
| + | The application actively used multiple CPU cores. | ||
| + | |||
| + | In an ideal case, a fully utilized four-core job running for ten seconds would show: | ||
| + | |||
| + | < | ||
| + | Elapsed | ||
| + | CPUTime | ||
| + | </ | ||
| + | |||
| + | <note tip> | ||
| + | A quick way to verify that your application is using multiple CPU cores is to compare '' | ||
| + | </ | ||
| + | |||
| + | <note tip> | ||
| + | Requesting multiple CPU cores with: | ||
| + | |||
| + | #SBATCH --cpus-per-task=4 | ||
| + | |||
| + | does not automatically make an application run in parallel. The application must also support multithreading and be configured to use the allocated cores. | ||
| + | </ | ||
| + | |||
| + | ===== Step 12: Submit a GPU Job ===== | ||
| To run a job on a GPU node: | To run a job on a GPU node: | ||
| Line 338: | Line 483: | ||
| #SBATCH --job-name=gpu-demo | #SBATCH --job-name=gpu-demo | ||
| #SBATCH --partition=gpu | #SBATCH --partition=gpu | ||
| - | #SBATCH --gpus=1 | + | #SBATCH --gres=gpu:1 |
| #SBATCH --cpus-per-task=4 | #SBATCH --cpus-per-task=4 | ||
| #SBATCH --mem=16G | #SBATCH --mem=16G | ||
| Line 352: | Line 497: | ||
| </ | </ | ||
| - | The command '' | + | The command '' |
| + | |||
| + | Users working within a project allocation may also need: | ||
| + | |||
| + | <code bash> | ||
| + | #SBATCH --account=alloc_xxxx_project | ||
| + | </ | ||
| + | |||
| + | ==== Interactive GPU Session ==== | ||
| + | |||
| + | To start an interactive shell on a GPU node: | ||
| + | |||
| + | <code bash> | ||
| + | srun -p gpu --gres=gpu: | ||
| + | </ | ||
| + | |||
| + | After the session starts: | ||
| + | |||
| + | <code bash> | ||
| + | nvidia-smi | ||
| + | </ | ||
| + | |||
| + | Interactive sessions are useful for testing software, debugging, and experimenting before submitting batch jobs. | ||
| ===== Useful Commands ===== | ===== Useful Commands ===== | ||
| Line 363: | Line 530: | ||
| | '' | | '' | ||
| | '' | | '' | ||
| + | | '' | ||
| | '' | | '' | ||
| | '' | | '' | ||
| Line 382: | Line 550: | ||
| * Submit jobs | * Submit jobs | ||
| * Monitor job execution | * Monitor job execution | ||
| - | * Review job outputs | + | * Verify resource utilization |
| - | * Request CPU and memory resources | + | |
| * Run Python applications | * Run Python applications | ||
| * Use project allocations | * Use project allocations | ||
| Line 390: | Line 557: | ||
| Continue with: | Continue with: | ||
| - | * [[hpc: | ||
| * [[hpc: | * [[hpc: | ||
| * [[hpc: | * [[hpc: | ||
| Line 408: | Line 574: | ||
| You are probably running the command on your local computer instead of the HPC login node. | You are probably running the command on your local computer instead of the HPC login node. | ||
| - | |||
| - | Verify your connection: | ||
| - | |||
| - | <code bash> | ||
| - | hostname | ||
| - | </ | ||
| ==== Invalid account or account/ | ==== Invalid account or account/ | ||
| Line 422: | Line 582: | ||
| * The specified allocation does not exist. | * The specified allocation does not exist. | ||
| * You do not have permission to use the specified allocation. | * You do not have permission to use the specified allocation. | ||
| - | |||
| - | Wait a few minutes after your first login or contact the project administrator. | ||
| ==== Job remains in PENDING state ==== | ==== Job remains in PENDING state ==== | ||
| Line 434: | Line 592: | ||
| squeue -u $USER | squeue -u $USER | ||
| </ | </ | ||
| - | |||
| - | More information can be found in: | ||
| - | |||
| - | * [[hpc: | ||
| ==== GPU job does not start ==== | ==== GPU job does not start ==== | ||
| Line 444: | Line 598: | ||
| * The job uses '' | * The job uses '' | ||
| + | * The job requests GPUs using '' | ||
| * GPU resources are available | * GPU resources are available | ||
| * Your allocation includes GPU access | * Your allocation includes GPU access | ||
| Line 460: | Line 615: | ||
| - [[hpc: | - [[hpc: | ||
| - | - [[hpc: | ||
| - [[hpc: | - [[hpc: | ||
| + | - [[hpc: | ||
| - [[hpc: | - [[hpc: | ||
| - [[hpc: | - [[hpc: | ||
en/hpc-new/getting_started.1795096929.txt.gz · Last modified: by eduardas
