Avoiding Quota issues on ORC Clusters

In this document we identify best practise for moving files to shared directories on ORC clusters.

Over quota errors are common problems encountered by users who have access shared project or group directories. These errors are generally triggered when copying files into a shared project or group directory. The quota on such storage locations is enforced by group ownership. Only the group that owns the directory has any quota to create files in the directory, so attempting to create any file or directory not owned by that group will fail with a 'Disk quota exceeded.' error.

The shared directories should all have the setgid bit set on their permissions to enforce inheritance of the group ownership, so files and directories created using the standard unix commands should be automatically created with the correct group ownership. The are cases however where some commands may try to override the default behaviour and preserve an existing ownership.

Avoid using the `mv` command to move files to a shared directory.

The mv comand when used across file systems effectively uses the cp command to copy the file followed by an rm command to remove the original file. The problem is that the cp command is effectivly run with the -a option (see below) and there is no way to change this behaviour.

The CP command

With the -a option cp will attempt to preserve all metadata associated with the original file, such as mode, ownerships, timestamps, extended attributes etc.

In place of -a when copying to a shared directory, try using the parameter: --preserve=mode,timestamps,links,xattr there is a handy short cut option -p for this option.

The RSYNC command

A frequently used command option used with rsync is -a. This is the so-called archive option and it is a shorthand for a list of other options -rlptgoD. Users should avoid using -a, and instead use -rlptoD which is equivalent apart from omiting the -g option which would enforce the preservation of group ownership.

The TAR command

To force tar to use a different group from the default group of the user or to override the group ownership of the files in an archive users may use the parameter --group="group-name"

Examples:

[aneil2@hopper2 ~]$ cp -rp /home/aneil2/New_Data /project/proj-aneil2/

[aneil2@hopper2 ~]$ rsync -v -rlptoD ~/New_Data /projects/proj-aneil2/