Migrating to New Scratch gs21

Recently, ICER announced the release of the new gs21 scratch system (see the announcement here) AND the retirement of ls15 and gs18 scratch space (sometimes referred to as “old scratch space” below). 

We would like to bring attention to the following:

(1) Highlight the important dates mentioned in the announcement:

June 2, 2022: Announcement of the gs21 release and ls15 and gs18 retirement is published. /mnt/gs21/scratch/<netID> is available for users to access. 
July 15, 2022: /mnt/scratch and the $SCRATCH variable will be updated to point to the new gs21 space.
July 22, 2022: ls15 and gs18 will be “read-only.”
July 31, 2022: ls15 and gs18 will be removed from service.
(2) Who should take action on this announcement?

(a) Any user currently using the scratch space at ls15 and gs18, either for storing files or running programs, should take action to migrate to gs21. To ensure that you are not using ls15 nor gs18 for file storage, please run the “quota” command on any dev node. If your quota usage of scratch space is not empty, see (3) for the actions you need to take.

(b) If you are the PI of any research group account, you should have received an email asking you to migrate your research group’s scratch space. By default, when a research group account is created, a scratch space for that research account is created as well. Unfortunately, the “quota” command does not show the usage of scratch space for the research group. The PI of the research group should check if there are any files stored at the scratch space of the research group at “/mnt/gs18/scratch/groups/<research_group_name>” and “/mnt/ls15/scratch/groups/<research_group_name>” and take action as explained in (3). 

(3) What actions should a user take?

(a) There are two things a user needs to do to migrate scratch space to the new gs21:

Copy all the files and directories you still need for your research from ls15 and gs18 to gs21. You do not have to copy over the files you do not need anymore. There is the same quota enforcement on gs21 (50 TB in the volume and 1 million in the number of files). Any methods a user is familiar with to transfer files from one location to another should work for this action (“cp”, “rsync”, etc.). NOTE: please decide how to organize the new storage before migrating. 
If you run jobs from scratch space or use scratch space for input or output files, you also need to modify your job script files if there are any file names or PATH settings pointed to ls15 or gs18, such as using explicitly “/mnt/ls15” or “/mnt/gs18” or implicitly “$SCRATCH”, “SC15” and “SC18”, etc. in it. After you have decided on the structure of your new scratch space--in other words, the new path of the files copied from the old space is determined--you can modify the corresponding path. Give the modified script a test run to see if there is any problem. 
(b) If you are the PI of any research group account, you should manage the migration of the scratch space of the research group to the new gs21 scratch space. Similar to the actions on migrating user scratch space mentioned in (a), the migration of the research group’s scratch space also involves copying files over to the new space and possibly modifying the script files if anyone was running a program from the old scratch space. In addition, due to the “shared” feature of the group space, extra consideration on whether to preserve the original ownership and access permissions of files may need to be taken when copying files to the new scratch space. 

(4) Due to the high utilization of gs18 and the age of ls15 and gs18, we are encouraging users to begin migrating to gs21 at their earliest convenience. Here is a list of important dates for reference:

June 2, 2022: Start your migration work mentioned in (3). Note that moving large amounts of data takes a long time. 
July 15, 2022: Finish modification of job script files and/or PATH settings. The old setting of /mnt/scratch and the $SCRATCH variable will no longer work. Make sure the path is correctly changed to the new gs21 location. Stop running jobs on old scratch space. 
July 22, 2022: The jobs that run on old scratch space may fail because the old scratch is "read-only."
July 31, 2022: Old scratch space is no longer accessible. No more action can be taken. 
(5) Tips and tricks for migration

As mentioned in the announcement, users who have large files (>10 TB ) may open a ticket for help moving the files. 
If you have been using both ls15 and gs18 and some files or directories have the same or similar names, you may want to first create a directory, named “LS15” for example, under /mnt/gs21/scratch/<your_NetID>, then copy all the files under ls15 scratch to /mnt/gs21/scratch/<your_NetID>/LS15/. Keep the structure of your scratch in gs21 the same as that in gs18. In this way, the modification of the file path and PATH settings may be easier and cleaner.  
If you have a large number of files to copy over, you may run the copy command as a slurm batch job so that you do not need to attend to it interactively.
As mentioned in the announcement, there are 200 TB of high-speed flash in the new system. It can be found at /mnt/gs21/flash/<NetID>/ with quotas here up to 1 TB for a user. This space is good for running jobs with a high frequency of I/O operations. We plan to have the flash storage auto-migrate data to the scratch storage; however, we do not yet have this functionality available. At this point, it is up to the user to manage the usage of this space. Due to size limits, we would appreciate it if users could only use it for running high-frequency I/O operations, and not for storing files with low frequency of I/O operations. 
If you are familiar with Globus file transfer, you may use Globus for transferring files to the gs21 scratch system. Globus is very robust in transferring large files.   

Xiaoge Wang, Ph.D. 
ICER Research Consultant