en:hpc
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
en:hpc [2022/04/15 06:28] – grikiete | en:hpc [2022/07/04 08:36] – [Batch Processing of Tasks (SLURM)] grikiete | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Description of the Equipment ====== | ====== Description of the Equipment ====== | ||
- | A Distributed | + | A High Performance |
- | VU MIF PST consists of a supercomputer from the clusters (the first number is the actual and available amount): | + | VU MIF HPC consists of a supercomputer from the clusters (the first number is the actual and available amount): |
^Title ^Nodes ^CPU ^GPU ^RAM ^HDD ^Network ^Notes| | ^Title ^Nodes ^CPU ^GPU ^RAM ^HDD ^Network ^Notes| | ||
Line 22: | Line 22: | ||
With the command [[https:// | With the command [[https:// | ||
- | With singularity you can prepare your container, for example: | + | You can prepare your container |
<code shell> | <code shell> | ||
$ singularity build --sandbox /tmp/python docker:// | $ singularity build --sandbox /tmp/python docker:// | ||
Line 45: | Line 45: | ||
There are ready-made scripts to run your **hadoop** tasks using the [[https:// | There are ready-made scripts to run your **hadoop** tasks using the [[https:// | ||
- | With [[https:// | + | With [[https:// |
<code shell> | <code shell> | ||
Line 60: | Line 60: | ||
====== Registration ====== | ====== Registration ====== | ||
- | The PST can only be used by registered users of the VU MIF computer network. Existing | + | |
- | Registracijai reikia užpildyti | + | * **For users of the VU computer network** - you must fill in the [[https://forms.office.com/ |
+ | |||
+ | * **For other users (non-members of the VU community)** - you must fill in the [[https:// | ||
+ | |||
+ | ====== Connection ====== | ||
+ | |||
+ | You need to use SSH applications (ssh, putty, winscp, mobaxterm) and Kerberos or SSH key authentication to connect to **HPC**. | ||
+ | |||
+ | If **Kerberos** is used: | ||
+ | |||
+ | * Log in to the Linux environment in a VU MIF classroom or public terminal with your VU MIF username and password or login to **uosis.mif.vu.lt** with your VU MIF username and password using **ssh** or **putty**. | ||
+ | * Check if you have a valid Kerberos key (ticket) with the **klist** command. If the key is not available or has expired, the **kinit** command must be used. | ||
+ | * Connect | ||
+ | |||
+ | If **SSH keys** are used (e.g. if you need to copy big files): | ||
+ | * If you don't have SSH keys, you can find instructions on how to create them in a Windows environment **[[duk:ssh_key|here]]** | ||
+ | * | ||
+ | * | ||
+ | |||
+ | The **first time** you connect, you **will not** be able to run **SLURM jobs** for the first **5 minutes**. After that, SLURM account will be created. | ||
+ | |||
+ | ====== Lustre - Shared File System ====== | ||
+ | |||
+ | VU MIF HPC shared file system is available in the directory '' | ||
+ | |||
+ | The system creates directory ''/ | ||
+ | |||
+ | The files in this file system are equally accessible on all compute nodes and on the **hpc** node. | ||
+ | |||
+ | Please use these directories only for their purpose and clean them up after calculations. | ||
+ | |||
+ | ====== HPC Partition ====== | ||
+ | |||
+ | ^Partition ^Time limit ^RAM | ||
+ | ^main | ||
+ | ^gpu ^48h | ||
+ | ^power | ||
+ | |||
+ | The time limit for tasks is **2h** in all partitions if it has not been specified. The table shows the maximum time limit. | ||
+ | |||
+ | The **RAM** column gives the amount of RAM allocated | ||
+ | |||
+ | ====== Batch Processing of Tasks (SLURM) ====== | ||
+ | |||
+ | To use computing resources of the HPC, you need to create task scenarios (sh or csh). | ||
+ | |||
+ | Example: | ||
+ | |||
+ | <code shell mpi-test-job.sh> | ||
+ | # | ||
+ | #SBATCH -p main | ||
+ | #SBATCH -n4 | ||
+ | module load openmpi | ||
+ | mpicc -o mpi-test mpi-test.c | ||
+ | mpirun mpi-test | ||
+ | </ | ||
+ | |||
+ | After submission and confirmation of your application to the ITOAC services, you need to create a user at https://hpc.mif.vu.lt/. The created user will be included in the relevant project, which will have a certain amount of resources. In order to use the project resources for calculations, | ||
+ | |||
+ | <code shell mpi-test-job.sh> | ||
+ | # | ||
+ | #SBATCH --account=alloc_xxxx_projektas | ||
+ | #SBATCH -p main | ||
+ | #SBATCH -n4 | ||
+ | #SBATCH --time=minutes | ||
+ | module load openmpi | ||
+ | mpicc -o mpi-test mpi-test.c | ||
+ | mpirun mpi-test | ||
+ | </ | ||
+ | |||
+ | |||
+ | Jame kaip specialūs komentarai yra nurodymai užduočių vykdytojui. | ||
+ | |||
+ | -p short - į kokią eilę siųsti (main, gpu, power). | ||
+ | |||
+ | -n4 - kiek procesorių rezervuoti (**PASTABA: | ||
+ | |||
+ | Užduoties pradinis einamasis katalogas yra dabartinis katalogas (**pwd**) prisijungimo mazge iš kur paleidžiama užduotis, nebent parametru -D pakeistas į kitą. Pradiniam einamajam katalogui naudokite PST bendros failų sistemos katalogus **/scratch/ | ||
+ | |||
+ | Suformuotą scenarijų siunčiame su komanda sbatch | ||
+ | |||
+ | '' | ||
+ | |||
+ | kuri gražina pateiktos užduoties numerį **JOBID**. | ||
+ | |||
+ | Laukiančios arba vykdomos užduoties būseną galima sužinoti su komanda squeue | ||
+ | |||
+ | '' | ||
+ | |||
+ | Su komanda scancel galima nutraukti užduoties vykdymą arba išimti ją iš eilės | ||
+ | |||
+ | '' | ||
+ | |||
+ | Jeigu neatsimenate savo užduočių **JOBID**, tai galite | ||
+ | |||
+ | '' | ||
+ | |||
+ | Užbaigtų užduočių **squeue** jau neberodo. | ||
+ | |||
+ | Jeigu nurodytas procesorių kiekis nėra pasiekiamas, | ||
+ | |||
+ | Vykdomos užduoties išvestis (**output**) yra įrašoma į failą **slurm-JOBID.out**. Jei nenurodyta kitaip, tai ir klaidų (error) išvestis yra įrašoma į tą patį failą. Failų vardus galima pakeisti su komandos **sbatch** parametrais -o (nurodyti išvesties failą) ir -e (nurodyti klaidų failą). | ||
+ | |||
+ | Daugiau apie SLURM galimybes galite paskaityti [[https:// | ||
- | Su suteiktu (pasirinktu) naudotojo vardu ir savo įvestu slaptažodžiu įgyjama teisė jungtis prie serverio **uosis.mif.vu.lt**, | ||
- | Adresu [[https:// | ||
en/hpc.txt · Last modified: 2024/02/21 12:50 by rolnas