Skip to content

Storage Space on the Cluster

Storage on the Cluster

Once you have your account set up on the ORC clusters, you will have access to the following storage options.

Home

/home/UserID
The /home directory is where you land when you first start a shell session on the cluster. From /home, you can navigate to the other spaces, including /scratch and /projects.

Properties of /home:

  • Read/write on the login/head and compute nodes
  • Limited to 60 GB per user. You will get an email warning to clear space once you approach this limit.
  • Backed up.
Restoring files from backup snapshots

/home is the only filesystem that is backed up. You can recover files from backup snapshots of /home directories by going to the directory in question and entering cd .snapshot

For example, to recover a file called run.slurm in the test directory from a snapshot named Weekly__2024-09-24_07_00_00_UTC, the process would look like this:

netid@hopper[~/]  cd test 
netid@hopper[~/test]  cd .snapshot
netid@hopper[~/test/.snapshot]  ls -ltr
total 0
drwxr-xr-x 2 netid users 4096 Jan 24  2022 Weekly__2024-10-22_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 Weekly__2024-10-15_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 Weekly__2024-10-08_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 Weekly__2024-10-01_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 Weekly__2024-09-24_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 Weekly__2024-09-17_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 Weekly__2024-09-10_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 Weekly__2024-09-03_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 Weekly__2024-08-27_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 monthly__2024-10-06_16_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 monthly__2024-09-06_16_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 monthly__2024-08-07_16_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 monthly__2024-07-08_16_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 monthly__2024-06-08_16_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 monthly__2024-05-09_16_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 daily__2024-10-22_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 daily__2024-10-21_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 daily__2024-10-20_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 daily__2024-10-19_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 daily__2024-10-18_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 daily__2024-10-17_07_00_00_UTC
drwxr-xr-x 2 netid users 4096 Jan 24  2022 daily__2024-10-16_07_00_00_UTC

netid@hopper[~/test/.snapshot]  cp Weekly__2024-09-24_07_00_00_UTC/run.slurm ~/test/

SCRATCH

/scratch/UserID

Properties of /scratch:

  • Read/write on every node - jobs can write output here
  • Scratch directories have no space limit
  • It is optimized for performance. Therefore, you should run I/O intensive calculations from this space. We encourage users to copy their data here, run their calculations and copy the results back to a more persistent space (/home, /projects, /groups).
  • As the name implies, this is a temporary space. Files in /scratch whose last modification date is more than 90 days will get deleted on the first day of every month. Therefore, make sure you move files from this space to a more persistent space (/home, /projects, /groups). Please see the FAQ for more detail.
  • /scratch is not backed up.

File purges based on file modification times

Files in /scratch modified more than 90 days ago are purged on the first day of every month. Sometimes, files you downloaded sooner than 90 days ago may get deleted because they retain the modification date they had when they were originally archived into a tarball. Therefore, we strongly encourage keeping backups of critical data outside the /scratch directory.

PROJECTS

/projects/project-owner
This is additional persistent free storage for projects whose storage requirements exceed the 60 GB that is available in the /home directory. Faculty can request 1 TB of shared storage space for their group. They can request expansion of this storage space based on the number of PhD students and postdocs in their group that are avtively using the HPC cluster and shared storage space. Students, postdocs and collaborators should request access to available projects spaces via their PI/advisor/supervisor. To request a projects directory, the faculty (PI) should send an email to orchelp@gmu.edu asking for the projects directory to be created. They should also provide a (user ID) list of those that should be granted access to the /projects directory. If students/other group members need to be added to an existing /projects space, the PI should send an email to orchelp@gmu.edu asking that hte new members be added to the direvtory.

The /projects space is

  • On ARGO, read-write on all nodes.
  • On HOPPER, read-write on all the nodes.
  • Not backed up.

The /projects space is a free storage location optimized for data that is actively used for computation on the HPC clusters. If you need more storage space on the cluster, please send an email to orchelp@gmu.edu to discuss available paid storage options such was /groups space described below..

GROUPS

/groups/group-name
This is storage that can be purchased on our MEMORI system by faculty members.They will need to provide an ORG code to charge the cost to and sign an SLA.

The current rate is $60/TB/year, storage space leased at this price will be valid for 5 years.

  • Storage is provided at 1 TB increments and can not be purchased in smaller chunks. Also, the cost can not be pro-rated regardless of the time of year it was purchased.
  • The rate of $60/TB/year is a yearly rate and the storage cost will be charged every year. Please budget for each year.
  • Facilities & Administrative (F&A) rates will be applied to the storage costs for all proposal storage awards. Please include the F&A cost in the budget.
  • F&A charges are not levied if “Indirect” funds are used to pay the storage cost.
  • The storage space is to be used to store and share research data.
  • ORC does not backup the data. We use data protection schemes like replication or erasure coding to prevent data loss due to hardware failure, but can not recover data that was intentionally or unintentially removed or changed.

Datasets

ORC offers a specialized directory for shared datasets in /datasets. If your work requires a publicly available dataset which is likely to be used by multiple users, you may request that the dataset be added to the datasets directory by contacting orchelp@gmu.edu.

Monitoring Storage Usage

When the storage space is exceeded, you will likely receive a 'Disk quota exceeded' error and you will not be able to write to the cluster. To avoid exceeding your /projects or /home quota limit, you can track your usage. You can quickly get storage disk usage for directory $DIR using the following options, in order decreasing speed

gdu (fast)

gdu --si -s $DIR

ncdu

ncdu --si $DIR

du (slow)

du --si -s $DIR

You will be unable save anything to your /projects or /home directory until you make space by

  • removing unnecessary files from /projects or /home
  • compressing files in /projects or /home
  • requesting and purchasing /groups storage at a cost of $50/TB/year ($60/TB/year starting FY24) by emailing orchelp@gmu.edu
  • moving files to your /scratch directory. Please note that /scratch is not backed up and it is subject to a 90-day purge policy.