The VU MIF HPC cluster uses the Lustre shared file system to store user data.
Unlike a personal computer, HPC jobs may run on different compute nodes. Lustre provides a shared storage space that is accessible from the login node and all compute nodes, allowing your jobs to access the same files regardless of where they are executed.
Lustre is a high-performance distributed file system designed for HPC environments.
It provides:
Each HPC user receives a home directory:
/scratch/lustre/home/username
where username is your HPC username.
You can quickly navigate to your home directory:
cd
or
cd ~
Display your current location:
pwd
When a job is submitted through SLURM, it may run on any available compute node.
Files stored on your local computer are not visible to cluster jobs.
Files stored in Lustre are available everywhere:
<note tip> Store all job scripts, input data, and output files in Lustre. This ensures they will be accessible wherever your job runs. </note>
A good practice is to create a separate directory for each project.
Example:
mkdir -p ~/projects/my-first-project cd ~/projects/my-first-project
You can then store:
in a single location.
Example:
my-project/ ├── data/ ├── scripts/ ├── jobs/ ├── results/ └── logs/
Example commands:
mkdir data scripts jobs results logs
List files:
ls
List files with details:
ls -lh
Show hidden files:
ls -la
Display the size of the current directory:
du -sh .
Display the size of your home directory:
du -sh ~
Display usage of subdirectories:
du -sh *
Copy a file:
cp input.txt backup.txt
Copy a directory:
cp -r project project_backup
Move a file:
mv old.txt new.txt
Delete a file:
rm file.txt
Delete a directory:
rm -r directory
<note important>
Be careful when using rm.
Deleted files cannot usually be recovered. </note>
Files can be transferred using:
Upload from your computer:
scp file.txt username@hpc.mif.vu.lt:~
Download to your computer:
scp username@hpc.mif.vu.lt:~/file.txt .
A job starts in the directory from which it was submitted.
Example:
pwd
sbatch job.sh
If the current directory is:
/scratch/lustre/home/username/my-project
the job will start in that location.
This is why it is important to submit jobs from your project directory.
Create a simple job:
#!/bin/bash #SBATCH --partition=main #SBATCH --output=where.out pwd hostname
After the job finishes:
cat where.out
Example output:
/scratch/lustre/home/username/my-project compute12
Notice that the job runs on a compute node but still accesses your Lustre files.
✔ Organize work into separate project directories
✔ Keep job scripts with the associated project
✔ Store important output files in Lustre
✔ Remove temporary files that are no longer needed
✔ Use meaningful file names
✔ Keep source code under version control (Git)
A compute node cannot access files stored on your laptop.
Always upload required files to Lustre before submitting a job.
Jobs start in the directory where they are submitted.
Verify your location before running:
pwd
Large intermediate files can quickly consume storage space.
Periodically review usage:
du -sh ~
Yes.
Files stored in Lustre are available from all login and compute nodes.
Yes.
Project directories and shared storage can be configured when needed.
Your files remain accessible because they are stored in the shared Lustre file system.
Only retain data that is still needed.
Unused data should be archived or removed to free storage resources.
Now that you know where HPC data is stored, learn how to submit more advanced jobs and request resources:
<note> This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, software stack, and operational policies. </note>