====== Shared File System (Lustre) ======
The VU MIF HPC cluster uses the Lustre shared file system to store user data.
Unlike a personal computer, HPC jobs may run on different compute nodes. Lustre provides a shared storage space that is accessible from the login node and all compute nodes, allowing your jobs to access the same files regardless of where they are executed.
===== What Is Lustre? =====
Lustre is a high-performance distributed file system designed for HPC environments.
It provides:
* Shared access from all cluster nodes
* High throughput for large datasets
* Storage for applications, scripts, and results
* Collaboration capabilities for research projects
===== Your Home Directory =====
Each HPC user receives a home directory:
/scratch/lustre/home/username
where ''username'' is your HPC username.
You can quickly navigate to your home directory:
cd
or
cd ~
Display your current location:
pwd
===== Why Use Lustre? =====
When a job is submitted through SLURM, it may run on any available compute node.
Files stored on your local computer are not visible to cluster jobs.
Files stored in Lustre are available everywhere:
* Login nodes
* Compute nodes
* Interactive sessions
* Batch jobs
Store all job scripts, input data, and output files in Lustre. This ensures they will be accessible wherever your job runs.
===== Creating a Project Directory =====
A good practice is to create a separate directory for each project.
Example:
mkdir -p ~/projects/my-first-project
cd ~/projects/my-first-project
You can then store:
* Input data
* Source code
* Job scripts
* Results
* Documentation
in a single location.
===== Typical Directory Structure =====
Example:
my-project/
├── data/
├── scripts/
├── jobs/
├── results/
└── logs/
Example commands:
mkdir data scripts jobs results logs
===== Checking Available Files =====
List files:
ls
List files with details:
ls -lh
Show hidden files:
ls -la
===== Checking Storage Usage =====
Display the size of the current directory:
du -sh .
Display the size of your home directory:
du -sh ~
Display usage of subdirectories:
du -sh *
===== Copying Files =====
Copy a file:
cp input.txt backup.txt
Copy a directory:
cp -r project project_backup
Move a file:
mv old.txt new.txt
Delete a file:
rm file.txt
Delete a directory:
rm -r directory
Be careful when using ''rm''.
Deleted files cannot usually be recovered.
===== File Transfers =====
Files can be transferred using:
* SCP
* SFTP
* WinSCP
* Visual Studio Code Remote SSH
Upload from your computer:
scp file.txt username@hpc.mif.vu.lt:~
Download to your computer:
scp username@hpc.mif.vu.lt:~/file.txt .
===== Using Lustre with SLURM =====
A job starts in the directory from which it was submitted.
Example:
pwd
sbatch job.sh
If the current directory is:
/scratch/lustre/home/username/my-project
the job will start in that location.
This is why it is important to submit jobs from your project directory.
===== Verify Where a Job Is Running =====
Create a simple job:
#!/bin/bash
#SBATCH --partition=main
#SBATCH --output=where.out
pwd
hostname
After the job finishes:
cat where.out
Example output:
/scratch/lustre/home/username/my-project
compute12
Notice that the job runs on a compute node but still accesses your Lustre files.
===== Best Practices =====
✔ Organize work into separate project directories
✔ Keep job scripts with the associated project
✔ Store important output files in Lustre
✔ Remove temporary files that are no longer needed
✔ Use meaningful file names
✔ Keep source code under version control (Git)
===== Common Mistakes =====
==== Storing Files Only on a Personal Computer ====
A compute node cannot access files stored on your laptop.
Always upload required files to Lustre before submitting a job.
==== Submitting Jobs from the Wrong Directory ====
Jobs start in the directory where they are submitted.
Verify your location before running:
pwd
==== Accumulating Large Temporary Files ====
Large intermediate files can quickly consume storage space.
Periodically review usage:
du -sh ~
===== Frequently Asked Questions =====
==== Can all compute nodes see my files? ====
Yes.
Files stored in Lustre are available from all login and compute nodes.
==== Can I share files with collaborators? ====
Yes.
Project directories and shared storage can be configured when needed.
==== What happens if my job runs on another node? ====
Your files remain accessible because they are stored in the shared Lustre file system.
==== Should I keep data after my project finishes? ====
Only retain data that is still needed.
Unused data should be archived or removed to free storage resources.
===== Next Step =====
Now that you know where HPC data is stored, learn how to submit more advanced jobs and request resources:
* [[hpc-new:slurm|Batch Job Execution (SLURM)]]
* [[hpc-new:resources|Resource Allocations and Project Requests]]
----
This guide was prepared and updated by the VU MIF HPC team with the assistance of Microsoft Copilot. All procedures and examples should be validated against the current HPC environment, software stack, and operational policies.