CCR IT staff will be migrating from the current EMC Isilon storage system to a new Vast Data all flash storage solution beginning on Monday, February 1, 2021. This document provides details of what users can expect for this migration and what changes you'll see after the migration is complete.
Background: Every 5 years we purchase a new storage system that provides CCR users enterprise-level redundant storage for permanent retention of their data. These storage platforms are where we maintain user home (/user) and shared group project (/projects) directories as well as all of our software installations (/util). The migration of data is always a difficult problem. Each time these migrations are done we have even more data to move. We begin the sync of data approximately one month before the date we switch the new storage into production. We use rsync and other tools to capture changes made on the current storage (adds/deletes/changes) and update them on the new storage. On the day we make the switch to the new storage, we block all logins and begin a final sync of /user and /projects directories to capture any changes that have been made in these directories since the last sync was run. Based on previous experience and the rates we're seeing currently in our sync processes, we anticipate this final sync to take at least 48 hours. Some project directories contain so many files that it's impossible to sync them in this amount of time. We have provided a detailed alert describing this year's plan.
What/When: This will be a a CENTERWIDE downtime - no users will be permitted to login to nearly all of CCR's resources beginning Monday, February 1 at 7am through (at least) Tuesday, February 2 at 5pm. Please see the detailed alert that includes our plans for any project directories that may not copy in that amount of time.
IMPORTANT NOTICE: Backups of Current Storage Filesystems Suspended as of 1/20/21
We have begun the transfer of backups to the new storage to remove the load off of the old storage system. After data is synced to the new storage it will be backed up. Please see our backup policy for details on our backup procedures and level of service.
What changes will there be with the new storage?
- Location of files: Users will still find their home directories in /user and their group shared directories in /projects. No name or permissions changes will be made.
- Change to quota enforcement: Currently we are utilizing user and group quotas but are switching to directory quotas with this new system. This may relieve users of the permissions problems some encounter in the shared directories; however, it could cause problems in the future with your group if you do not set permissions correctly. We HIGHLY recommend leaving the default group permissions on your subdirectories to allow all in your group access to these shared files and directories.
- Quota reporting: Users must specify the directory name to see the quota for /project and /panasas/scratch directories. Please see the iquota documentation for more info
- iquota -p /projects/academic/groupname
- iquota -p /projects/industry/groupname
- iquota -p /projects/rpci/groupname
- iquota -p /panasas/scratch/grp-groupname
- Default quotas: Home directory quotas are increasing from 5GB to 10GB.
- File limits: To protect the file system we've set a maximum file limit for home directories of 10 million files and for project directories of 200 million files. The iquota command will report total number of files you are currently storing in your directories.
- Backup policy change: Directories with more than 50 million files in them are no longer backed up. More details can be found in the backup policy. The PI owner of any project directory that meets this criteria will be notified about being removed from backups.
- Home directory permissions: Anyone with non-standard directory permissions on their home directory will have these changed to 700 (i.e. owner has full access, no one else has access). This is a system requirement for SSH keys when we upgrade to CentOS 8. Users who are affected by this will be contacted directly by CCR staff prior to the downtime. For reference, this is only a small number of CCR users. We have never recommended that users change their home directory permissions to share files. File sharing should always be done within the shared project directories.
- Allocations: For groups with a project directory, your ColdFront project will now display an allocation for this storage. The total number of TB (qty) on the allocation is your total quota including anything you've purchased.
- Performance: Users should find this all-flash based storage system much faster than the current system.
Coming with the next downtime (2/23):
- Panasas scratch directories will be switched to directory (volume) quotas to match the project directories and provide a consistent user experience.
- Panasas scratch allocations will be added to all ColdFront projects to allow for the automation of directories and quota setting.