IT wiki

VU MIF STSC

User Tools

Site Tools


en:hpc-new:getting_started

First Successful Job (15 Minutes)

This guide walks you through the basic HPC workflow:

  1. Connect to the HPC system
  2. Understand the difference between login and compute nodes
  3. Create and submit your first job
  4. Monitor job execution
  5. View job results
  6. Request CPU and memory resources
  7. Learn how project allocations work
  8. Run a multi-core Python application
  9. Verify CPU utilization
  10. Submit a GPU job

If you can complete this guide, you are ready to start running your own workloads on the HPC cluster.

Before You Start

Make sure that:

  • You have completed Registration
  • You have an active HPC account
  • You have an SSH client installed
  • You can connect to the HPC login node

Need access first?

Step 1: Connect to the Cluster

Open a terminal and connect to the HPC login node:

ssh username@hpc.mif.vu.lt

Replace username with your HPC username.

Verify that you are connected:

hostname
pwd

You should see the hostname of the login node and your home directory path.

<note important> Your SLURM account is created automatically after your first successful login. This process may take up to 5 minutes.

Until the account is created, job submission may fail with account-related errors. </note>

For detailed instructions see:

Step 2: Understanding Login Nodes and Compute Nodes

Unlike a personal computer, computational workloads are not normally executed directly from the login node. Instead, jobs are submitted to SLURM, which schedules them on available compute nodes.

The login node is intended for:

  • Editing files
  • Compiling code
  • Managing data
  • Preparing and submitting jobs

The compute nodes are used for:

  • CPU-intensive calculations
  • Simulations
  • Data analysis
  • Machine learning and AI workloads

<note important> Do not run computationally intensive applications directly on the login node. Submit them through SLURM so they can be executed on the appropriate compute resources. </note>

Step 3: Create a Working Directory

Create a directory for this tutorial:

mkdir getting_started
cd getting_started

Verify your location:

pwd

Step 4: Create Your First Job Script

Create a new file:

nano hello.sh

Paste the following content:

#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --partition=main
#SBATCH --output=hello.out
#SBATCH --time=00:01:00
 
echo "Hello from HPC!"
hostname
date

Save the file and exit the editor.

Step 5: Submit the Job

Submit the script to the scheduler:

sbatch hello.sh

Example output:

Submitted batch job 123456

The returned number is your Job ID.

Step 6: Monitor the Job

Check the status of your jobs:

squeue -u $USER

When the job disappears from the queue, it has finished.

Step 7: View the Results

Display the output file:

cat hello.out

Example output:

Hello from HPC!
compute01
Mon Nov 17 14:32:05 EET 2026

Congratulations! You have successfully executed your first HPC job.

Step 8: Request Resources for a Job

Real HPC workloads usually require specific CPU, memory, and runtime allocations. These resources are requested using SLURM directives.

Create a new script:

nano resources.sh

Paste the following content:

#!/bin/bash
#SBATCH --job-name=resources-demo
#SBATCH --partition=main
#SBATCH --cpus-per-task=4
#SBATCH --mem=4G
#SBATCH --time=00:05:00
#SBATCH --output=resources.out
 
echo "Job ID:"
echo $SLURM_JOB_ID
 
echo ""
echo "Partition:"
echo $SLURM_JOB_PARTITION
 
echo ""
echo "Allocated CPUs:"
echo $SLURM_CPUS_PER_TASK
 
echo ""
echo "Finished."

Submit the job:

sbatch resources.sh

Understanding Resource Requests

Directive Description
–partition=main Run the job in the standard CPU partition
–partition=gpu Run the job in the GPU partition
–cpus-per-task=4 Request 4 CPU cores
–mem=4G Request 4 GB of RAM
–time=00:05:00 Request up to 5 minutes of runtime
–output=resources.out Save output to a file

<note tip> Most CPU workloads should use the main partition.

GPU workloads should use the gpu partition. </note>

<note tip> If no runtime limit is specified, SLURM uses a default limit of 2 hours. However, specifying a realistic runtime is recommended because it helps the scheduler start jobs more efficiently. </note>

<note tip> Request only the resources you actually need. Smaller jobs generally start faster and help ensure fair sharing of cluster resources. </note>

Check the Job Output

Once the job completes:

cat resources.out

Step 9: Using Project Allocations

Some users are assigned project allocations that provide additional CPU, GPU, memory, or storage resources.

If your project administrator provides an allocation name, add it to your job script:

#!/bin/bash
#SBATCH --job-name=project-demo
#SBATCH --partition=main
#SBATCH --account=alloc_xxxx_project
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=01:00:00
 
hostname

The allocation name determines which project's resources are used to run the job.

<note important> VU MIF users using the default allocation do NOT need to specify the –account parameter.

Only users associated with a project allocation should use the –account directive. </note>

Example:

#SBATCH --account=alloc_xxxx_project

Step 10: Run a Multi-Core Python Job

Python is one of the most commonly used programming languages on HPC systems. This example performs a matrix multiplication, a common operation in scientific computing, machine learning, and data analysis.

Create a Python Script

Create a file named matrix.py:

nano matrix.py

Paste:

import os
import numpy as np
from time import perf_counter
 
cpus = os.environ.get("SLURM_CPUS_PER_TASK", "unknown")
 
print(f"CPU cores allocated by SLURM: {cpus}")
 
size = 2000
 
print(f"Creating two {size}x{size} matrices...")
 
a = np.random.rand(size, size)
b = np.random.rand(size, size)
 
start = perf_counter()
c = a @ b
end = perf_counter()
 
print("Matrix multiplication completed")
print(f"Result shape: {c.shape}")
print(f"Execution time: {end - start:.2f} seconds")

Create a SLURM Job Script

Create a new file:

nano python-job.sh

Paste:

#!/bin/bash
#SBATCH --job-name=python-demo
#SBATCH --partition=main
#SBATCH --cpus-per-task=4
#SBATCH --mem=2G
#SBATCH --time=00:05:00
#SBATCH --output=python-demo.out
 
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OPENBLAS_NUM_THREADS=$SLURM_CPUS_PER_TASK
export MKL_NUM_THREADS=$SLURM_CPUS_PER_TASK
 
python3 matrix.py

Submit the job:

sbatch python-job.sh

Check the Results

After the job finishes:

cat python-demo.out

Example output:

CPU cores allocated by SLURM: 4
Creating two 2000x2000 matrices...
Matrix multiplication completed
Result shape: (2000, 2000)
Execution time: 1.82 seconds

Optional: Load a Python Module

Depending on the cluster software environment, a specific Python version may need to be loaded first:

module avail Python

Load one of the available versions shown by the system:

module load Python/<version>

Verify:

python3 --version

Step 11: Verify CPU Usage

While a job is running, you can monitor resource utilization with:

sstat -j JOBID.batch --format=JobID,AveCPU,AveRSS,MaxRSS

Example:

       JobID     AveCPU     AveRSS     MaxRSS
------------ ---------- ---------- ----------
123456.batch   00:00:01      500M      620M

Where:

Field Description
AveCPU Average CPU time consumed by job tasks
AveRSS Average memory usage
MaxRSS Maximum memory usage

You can identify your Job ID with:

squeue -u $USER

Compare Elapsed Time and CPU Time

After the job finishes:

sacct -j JOBID --format=JobID,AllocCPUS,Elapsed,CPUTime

Example:

JobID          AllocCPUS    Elapsed    CPUTime
------------ ---------- ---------- ----------
123456                4   00:00:02   00:00:07
Field Description
AllocCPUS Number of allocated CPU cores
Elapsed Wall-clock runtime (real time)
CPUTime Total CPU time used by all allocated CPU cores

How to Interpret the Results

Single-threaded application:

AllocCPUS = 4
Elapsed   = 00:00:10
CPUTime   = 00:00:10

The application used approximately one CPU core, even though four cores were allocated.

Multi-threaded application:

AllocCPUS = 4
Elapsed   = 00:00:10
CPUTime   = 00:00:38

The application actively used multiple CPU cores.

In an ideal case, a fully utilized four-core job running for ten seconds would show:

Elapsed   = 00:00:10
CPUTime   = 00:00:40

<note tip> A quick way to verify that your application is using multiple CPU cores is to compare CPUTime with Elapsed. If CPUTime is several times larger than Elapsed, the workload is utilizing multiple CPU cores concurrently. </note>

<note tip> Requesting multiple CPU cores with:

#SBATCH –cpus-per-task=4

does not automatically make an application run in parallel. The application must also support multithreading and be configured to use the allocated cores. </note>

Step 12: Submit a GPU Job

To run a job on a GPU node:

nano gpu-demo.sh

Paste:

#!/bin/bash
#SBATCH --job-name=gpu-demo
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=01:00:00
 
nvidia-smi

Submit the job:

sbatch gpu-demo.sh

The command nvidia-smi displays information about the GPU allocated to the job.

Users working within a project allocation may also need:

#SBATCH --account=alloc_xxxx_project

Interactive GPU Session

To start an interactive shell on a GPU node:

srun -p gpu --gres=gpu:1 --pty $SHELL

After the session starts:

nvidia-smi

Interactive sessions are useful for testing software, debugging, and experimenting before submitting batch jobs.

Useful Commands

Command Purpose
pwd Show current directory
ls List files
cd Change directory
sbatch job.sh Submit a job
squeue -u $USER View your jobs
sacct -j JOBID View completed job information
sstat -j JOBID.batch Monitor resource usage of a running job
scancel JOBID Cancel a job
cat file Display file contents

Typical Resource Requests

Workload Partition CPUs Memory GPUs
Learning and testing main 1-2 1-4 GB None
Data analysis main 4-8 8-32 GB Optional
Scientific simulations main 8-64 16-128 GB Optional
Machine learning and AI gpu 4-16 16-64 GB 1 or more

Next Steps

You now know how to:

  • Connect to the HPC cluster
  • Create job scripts
  • Submit jobs
  • Monitor job execution
  • Verify resource utilization
  • Run Python applications
  • Use project allocations
  • Submit GPU jobs

Continue with:

Common Problems

Permission denied

Verify that:

  • Your account has been activated
  • You are using the correct username
  • Your SSH credentials are correct

sbatch: command not found

You are probably running the command on your local computer instead of the HPC login node.

Invalid account or account/partition combination specified

Possible causes:

  • Your SLURM account is still being created.
  • The specified allocation does not exist.
  • You do not have permission to use the specified allocation.

Job remains in PENDING state

Resources may currently be unavailable.

Check your jobs:

squeue -u $USER

GPU job does not start

Verify that:

  • The job uses –partition=gpu
  • The job requests GPUs using –gres=gpu:N
  • GPU resources are available
  • Your allocation includes GPU access

Python command not found

Verify that Python is installed in the environment or load the appropriate Python module:

module avail Python

Where to Go Next

Recommended learning path:


<note> This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, software stack, and operational policies. </note>

en/hpc-new/getting_started.txt · Last modified: by eduardas

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki