File systems and file transfer
Introduction
Warning
There are no backups of our file systems at this point. You have to make sure to backup your data yourself.
Please make sure only to store the data you need for your current work on the SC file systems. The clusters and their file systems are shared resources and all users need to share these resources.
Please make sure to delete your temporary local files you created during a computation, when your job finishes. You may use your job script or an epilog script to implement a clean-up procedure.
Quotas
We are now enfording quotas on the file systems.
You have 30 GB in your /home directory and up to 1 TB in your /work directory with a workspace application.
Please make sure to clean up your files and directories regularly.
File systems
On our cluster we have three different types of storage tiers that have varying levels of performance, capacity, and accessibility. These tiers are designed to provide a balance between speed and cost, as well as to accommodate different types of data and workloads.
Home directory
The Home directory ~ or /home/sc.uni-leipzig.de/<scuser>
This is a user's personal storage space where they can store their files, scripts, and configuration files.
It is accessible from any node on the cluster.
Work directory
The work directory /work is intended to be used with workspaces.
This is a directory where you as a users can store your large data files that are needed for your calculations.
It is accessible from any node on the cluster, which make it a perfect storage space to share large libraries of files that need to be shared with a group of users.
Warning
Beware that this a specialized parallel filesystem optimized for large file access! Reading and writing to many small files may not yield the expected performance and may even slow down the system for others users.
Local scratch directory
The local scratch /lscratch
The local scratch directory on our clusters is a fast storage space located on each individual compute node. It should be used to store temporary data generated during computation and is optimized for fast read and write access. However, it is small in size, not backed up, and should only be used for temporary data. Users are advised to move important data to the shared work directory or home directory.
Copy data from and to the SC systems
The easiest way to copy data to these file systems is to use the command line tool secure copy scp.
The following commands assume your are copying from your local workstation to the SC clusters.
To copy a single file to your work directory use
Complete directories can be copied recursively by adding the -r switch, e.g.
If you want to copy files/directories to your home directory use
Tip
Notice here that your home directory can be abbreviated to just a single tilde ~ character.
Let us assume your SC username is ab123defg.
~ would automatically be interpreted as /home/sc.uni-leipzig.de/ab123defg.
If you have your data on a Windows machine you can use winscp.