A Distributed Computing Network (DCN) is a specially designed network of computers capable of running applications that can exchange data efficiently.

VU MIF PST consists of a supercomputer from the clusters (the first number is the actual and available amount):

Total 40/41 nodes, 1912 CPU cores with 17TB RAM, 32 GPU with 1TB RAM.

The processor below = CPU = core - a single core of the processor (with all hyperthreads if they are turned on).

main and gpu are installed Qlustar 11 operating system (OS) with Linux core. It is created Ubuntu 18.04 LTS based. power is installed Ubuntu 18.04 LTS.

You can check the list of OS package with the command dpkg -l (in log in node hpc or in power nodes).

With the command singularity it is possible to make use of ready-made copies of container files in directories /apps/local/hpc, /apps/local/nvidia, /apps/local/intel, /apps/local/lang or to download from singularity and docker online directories. You can also create your own singularity containers using the MIF cloud service.

With singularity you can prepare your container, for example:

$ singularity build --sandbox /tmp/python docker://python:3.8
$ singularity exec -w /tmp/python pip install package
$ singularity build python.sif /tmp/python
$ rm -rf /tmp/python

Similarly, you can use R, Julia or other containers that do not require root privileges to install packages.

If you want to add OS packages to the singularity container, you need root/superuser privileges. With fakeroot, we simulate them, and copy the required library libfakeroot-sysv.so into the container, example:

$ singularity build --sandbox /tmp/python docker://ubuntu:18.04
$ cp /libfakeroot-sysv.so /tmp/python/
$ fakeroot -l /libfakeroot-sysv.so singularity exec -w /tmp/python apt-get update
$ fakeroot -l /libfakeroot-sysv.so singularity exec -w /tmp/python apt-get install python3.8 ...
$ fakeroot -l /libfakeroot-sysv.so singularity exec -w /tmp/python apt-get clean
$ rm -rf /tmp/python/libfakeroot-sysv.so /tmp/python/var/lib/apt/lists (you can clean up more of what you don't need)
$ singularity build python.sif /tmp/python
$ rm -rf /tmp/python

Kataloge /apps/local/bigdata yra paruošti scenarijai pasileisti savo hadoop užduotis pasinaudojant Magpie rinkiniu.

Su JupyterHub galite interneto naršyklės pagalba vykdyti skaičiavimus su python komandų eilute ir pasinaudoti JupyterLab aplinka. Jeigu savo namų kataloge instaliuosite savo JupyterLab aplinką, tai reikia instaliuoti papildomai batchspawner paketą - tada jums startuos jūsų aplinką, pvz:

$ python3.7 -m pip install --upgrade pip setuptools wheel
$ python3.7 -m pip install --ignore-installed batchspawner jupyterlab

Taip pat jūs galite pasinaudoti savo pasidarytu konteineriu per JupyterHub. Tame konteineryje reikia instaliuoti batchswapner ir jupyterlab paketus bei sukurti script'ą ~/.local/bin/batchspawner-singleuser su vykdymo teisėmis (chmod +x ~/.local/bin/batchspawner-singleuser)

#!/bin/sh
exec singularity exec --nv myjupyterlab.sif batchspawner-singleuser "$@"