Guidelines for choosing file storage and I/O on HPCC

THURSDAY, January 19th — HPCC provides five types of file storage. They are referred to as HOME, RESEARCH, SCRATCH, LOCAL and RAMDISK respectively in this document.  Each type of storage is different from either a hardware or software point of view. Figure 1 gives a brief overview of the file storage location with respect to a single “node”.  This article explains the differences between these storage types and the implications for accessibility, allocation and application use.

HOME space is the storage created for each user account. By default, it is where users login to HPCC (/mnt/home/$USER). This space is set to be only accessible by the user and is often referred to as “home directory”.  Each new user starts with a 50 GB limit for files. Users can request an increase in storage up to 1TB by completing the Quota Increase Request. Storage space greater than 1TB is available for an annual fee, paid through an MSU account number.

HOME uses the ZFS file system automatically backed-up hourly for the last 24 hours, daily for the last week, and weekly for the last month. To access the backup, please go to the directory ~/.zfs/snapshot/. See Restoring files from backup for more information. For the system security and user data privacy, a user should not open HOME directory access to others. If you report an issue with using the system, please attach files to ticket email for reference.  Do not point to your HOME space as iCER system administrators will not have access to your home directory.

RESEARCH space is created upon the request of a MSU faculty member and is provisioned with 1TB free storage. This space is associated with a group and accessible to all members that have been added to that group. It is convenient for sharing files within a group. To allow group members to access files stored in this space, ensure that umask and the appropriate file permissions are set. RESEARCH also uses ZFS file system so files stored here are backed up. ZFS supports “auto mount” capability which allows a directory to be “visible” (mounted to the node) only when it is explicitly requested. For example, if you’re logged into HPCC and want to go to your RESEARCH space, running “cd /mnt/research” followed by “ls” may not display your research space in the “ls” output. Don’t panic, simply run “cd GROUP”, where GROUP is the name of your RESEARCH space. This should bring you into your RESEARCH space. If not, please contact us for help. In general, you should directly go to your groups space by running “cd /mnt/research/GROUP”.

Table 1: This table gives a brief overview of the disk spaces for file I/O operations on HPCC. $USER is your HPCC username and GROUP represents the research group name.

SCRATCH space is a very large storage system on HPCC with capacity of 1.9PB. It uses open source Lustre software to manage heavy and frequent file I/O in parallel. It is highly scalable for a large number of nodes and large (size) files. The shortcoming is that it does not provide auto backup and data corruption may occur (although it is very rare). It is a good choice for running parallel programs; especially for programs running on a large number of nodes and/or programs with I/O access to large (size) files. Because there is no auto backup, users should use it as working space for temporary data file storage and save their results or data back to HOME or RESEARCH space. Do not use the SCRATCH space for long-term storage.  In addition, the Lustre file system is not good at handling high frequency I/O operations on large numbers of small files.  You may notice that your program is slow when running on SCRATCH space if your jobs try to create or access many small files.  An example of what to avoid when using SCRATCH includes: using the “tar” command to unpack a .tar file or running “tar” commands inside a loop within a script. This type of workload may create lots of directories and files at once causing a very heavy load on the Lustre system, which may cause damage to it. If you have any doubts about how to use SCRATCH, please contact us and we will be happy to work with you to find a solution.

LOCAL space is a hard drive on a node as shown in Figure 1. Files on this space are accessed locally on the node and do not need to go through network. For jobs using a single node or using multi-node with each process only accessing its own local file, this space is a good choice.  When network traffic is high, using this space will likely allow your program to run faster than running on HOME, RESEARCH or SCRATCH space. Please note that LOCAL space is not auto backup, is not accessible from other nodes, and is shared with all processes running on the same node, so it should be used as temporary storage for program execution. When program execution is complete, any useful files in this space should be saved back to your HOME or RESEARCH space.

RAMDISK space is only a “logical” storage. As shown in Figure 1, it sits inside a node’s RAM, not disk.  By default, Linux supports a system tool that intercepts the I/O requests to /dev/shm with memory operations.  We may think of RAMDISK as a virtual disk in memory. As such, access to this space is actually access to RAM. Since the bandwidth access to RAM is much higher, the I/O operation is much faster than from LOCAL space. However, since files occupy the node’s memory, the usable RAM space of the program is reduced. This space is good for programs which need a small amount of memory and require very frequent I/O on small files.

Table 1 above provides more detailed information for each type of storage. Please use the table below to choose which file system is better for your job. If you are not sure which file system to use, please ask an iCER research consultant or write your question to iCER contact site:

For more information regarding file systems and I/O operations visit the HPCC wiki pages below: