Blue Storage Data Migration FAQ

What notifications will we get and what does the group sponsor need to communicate to their users?

  1. An email was sent in June to both the hpc-announce list and to sponsors of all group/top-level directories on /ufrc. That email ensures all users are aware of the pending changes and encourages groups to plan for communication of the scheduled transition date once sent in the second email to the sponsor.
  2. The second email will go only to the sponsor of the group with a scheduled transition date one or more weeks prior to the scheduled date – multiple sponsors may be scheduled on the same day. If a sponsor agrees with the scheduled date, no response is necessary. 
  3. It is up to the sponsor to reply to if a different date is needed.
    1. A sponsor can suggest a different date if a reply to is received before the scheduled migration date.  UFRC will make every reasonable effort to satisfy the request (there are some data management and concurrency considerations when scheduling transition dates).
  4. If the date is acceptable, it is up to the sponsor to communicate this date with the group members. 
    1. If no reply is received, the migration will happen on the scheduled date.

What will change?

  • Path: Blue storage will be accessible via the path /blue/<group>.
  • Quotas will change from filesystem-wide group quotas to per-directory quotas.
    •  Each group has a directory and all files in that directory will count towards the group’s quota.
  • Permissions: /blue/<group> will be group-writable
    • UFRC will continue to create user and share subdirectories inside /blue/<group> as a suggested way to organize group data.  Groups can use them (or not) as they wish.
    • By default, top-level directories in the new Blue storage will not be world-readable.
  • Purchase the storage you need: The 2TB of free Blue storage will be discontinued – please see the announcement.
  • SMB access to the new Blue storage system will use \\\blue. 
    • SMB access to Orange storage will also be available via \\\orange in June 2020.  The existing Orange SMB access via \\\orange will remain available, pending a future announcement.  
    • Users should not attempt to access their data via \\\blue until their group has been notified that their transfer is complete.

How will data be copied to the new Blue storage system?

UFRC staff will employ a scalable copy tool called Lustre Data Mover on a group-by-group basis.  The copy tool will be used during routine maintenance windows to avoid impacting /ufrc interactive response during normal business hours. 

Users do not need to copy their own data.

Please clean up data that is no longer needed or valuable before the transfer from /ufrc to /blue is completed.

Will any data on /ufrc not be copied?

  • Only group directories on /ufrc associated with an active allocation will be copied to the new Blue storage system.
  • Some Unix “special” files (character device, socket, fifo, named pipes, etc.) will not be copied. They must be recreated on the new files system. Since these files are usually created by applications, that will be automatic in most cases.

Will POSIX ACLs be copied?

Note: POSIX ACLs are not used by most groups

POSIX ACLs found on directories at the /ufrc/<group> and /ufrc/<group>/<subdir> will be reimplemented recursively, but any implemented in further subdirectories will not. 

How will the transition to the new Blue storage system work?

Rolling group-by-group transitions between June and August 2020 will be scheduled, during which time a sponsor’s group directory will be synchronized to the new system and be made available as /blue/<group>.  The original /ufrc/<group> directory will no longer be available.  No data from /ufrc will be removed as part of this process. 

During the transition all users of the group must be logged out of HiPerGator and all jobs stopped.

What happens on the day of transition?

The transition will occur anytime within the 24-hour period on the scheduled date.  

Any logins (ssh/sftp/globus/smb) will be killed, and any jobs in the batch system (pending or running) will be removed.  

The sponsor will receive an email notifying you that the transition is complete, and that the group’s users should now access your Blue storage via /blue/<group>. Again, it will be up to the sponsor to notify group members that the transition is complete.

Will burst quotas be transferred to /blue storage?

Yes, burst quotas will transfer as currently set.