Storage

  1. HiPerGator provides storage systems for keeping data; several storage types are available, each with different characteristics. The goal is to make things simple to use but still efficient for high performance when needed, as well as making storage cost effective for research budgets.
  2. Storage is allocated by quota assigned to groups. The users in a group share the storage quota; when the quota limit is reached, no more data can be written.
  3. The system monitors the storage use of all groups. Members of a group will receive an email when the use of storage exceeds 90% of the quota limit; the email contains the list of all users in the group and their use, so that groups can decide how to proceed and avoid the crisis of not being able to write any data. The options are to delete some data that is no longer needed, or to buy extra storage which will increase the quota limit.
  4. The storage information is also available on the web portal, which requires authentication with GatorLink. To be implemented by Dec 2017.
  5. The types of data are the following, with details described below:
    1. Storage for keeping data with high performance access
      • Investors need to buy an allocation of storage for the group to keep all data in excess of the 2 TB that is available at no cost to any investor of compute cores. When the amount of storage used reaches the limit of storage allocated, it is no longer possible to write more data.
      • An example is data that is collected by an instrument, such as a satellite image or brain scan. The data is large and is the starting point for computational work.
      • Data is not backed up, unless a backup service is purchased. This is a separate service described below.
      • The underlying hardware in the storage systems does have RAID protection against disk drive failures. This is not the same as data backup.
    2. Storage for scratch and ephemeral data with high performance access
      • Groups with investments in large number of NCUs can get temporary quota to run jobs that create large amounts of temporary data that is deleted at the end of the job. If the data is not deleted, then it will not be possible to write new data after the grace period of one month expires. The notification system will help groups to manage their use of storage.
      • The underlying hardware does have RAID protection against disk drive failures, which is different than data backup.
    3. Storage for keeping data at low performance access.
      • An example is a collection of images that have been processed and no longer need to be accessed for high-performance data analysis and processing, but still need to be accessed occasionally.
      • This option is not available yet. The implementation is expected in second half of 2017.
    4. Storage with copy of the data for disaster recovery
      • Groups can buy storage that is copied such that a snapshot of the data is taken once per day and kept for 30 days. This will allow recovery from some data loss if discovered within 30 days.
      • The copy is not intended as a backup with historical record of versions.
      • Storage with this feature can be purchased for the high performance or low performance category.
      • The current system for replicated storage /rlts will continue to be supported for existing users, but will not accept new users. New requests for replicated storage will be implemented on /ufrc with replication done to the UFIT TSM tape robot.
    5. Tape backup service
      • In addition to the above options, researchers can by tape backup service for all data in a directory that they specify.
      • To use this service, a separate account must be supplied to cover the monthly charges.
      • The TSM STANDARD schedule is as follows:
        • All data that has not been copied to tape, because it is new or a modified version, will be copied to tape during the nightly backup cycle.
        • Tapes are replicated with a copy sent off-site to Tallahassee, FL, or Atlanta, GA.
        • For existing data 7 versions are kept; after data is deleted from the source storage, only 5 versions are kept.
        • Older versions of existing data are kept up to 60 days after the last copy is written.
        • The last version of a deleted file is deleted after 90 days.
      • Different schedules can be configured as needed.