Turbo Storage User Guide

Overview

Turbo is a high-capacity, fast, reliable and secure data storage service for researchers across the University of Michigan.

It is configured to be easily sharable with on-campus resources (such as the Great Lakes HPC cluster) as well as with off-campus systems and collaborators. Researchers can learn how to purchase storage space via the Turbo Getting Started page and determine the option that best fits their needs. More details can be found on the Turbo Configuration page.

This user guide is intended to provide basic information and troubleshooting assistance for Turbo Research Storage. However, if at any point you have problems or questions, feel free to contact us at [email protected].  Please include the following information when contacting us:

1) Name of Turbo Volume   2) Hostname or IP Address of your System   3) Client Operating System

What is Turbo?

Turbo is a high-capacity, high-performance network storage solution, providing secure and reliable data storage to users across the University of Michigan. Available to our High Performance Computing Clusters (such as Great Lakes, Lighthouse, and Armis2) as well as to local hosts in labs (or at home), Turbo enables our researchers to access and leverage their data quickly and easily.

Turbo is also designed to be highly flexible. While it is tuned for large files (roughly 1 megabyte in size or larger), it is still capable of efficiently storing small files, such as: word documents, spreadsheets, image files, etc. Additionally, Turbo supports the storing of sensitive data on HIPAA compliant systems, such as the Armis 2 cluster managed by ARC.

Why Use ARC Network Storage?

Now, you may be asking yourself: I can get Hard Drives much cheaper than Turbo Storage, why should I use ARC Network Storage?

  • Reliability — Turbo offers numerous levels of data protection to ensure the accessibility and integrity of user data.
  • Accessibility — Turbo storage can be accessed from multiple devices and multiple locations simultaneously.
  • Flexibility — Turbo can accommodate multiple file types of various sizes.
  • Ease of Management — changes to storage size, the addition or removal of snapshots, and a many other options can easily be configured.

Storage Resource Software

If you are unsure which of our storage services should be used to host your data, we have written some software that you can download and execute to analyze your files to understand how much of your data is stored in large files, how much of your data has been accessed recently, and the distribution of file sizes and access times. Download the software from the Storage Guidance Tool.

This software doesn’t examine the contents of any data files, it merely scans file attributes, it also does not store any file names after searching through the filesystem.

If you have any questions on this software or if you are unsure about any of the recommendations the tool sends you, please contact us at [email protected] for assistance.

Turbo Access Methods

The Turbo Research Storage, is what’s called “Network Attached Storage” or NAS for short. A NAS is simply a storage device that is available to users over the Network. So, if your computer is on the University of Michigan Network (or U-M VPN), you can access Turbo! The wonderful thing about a NAS, is that it requires no formatting or disk management by the User! A NAS will provide File Level access to storage, just as if you were browsing through the documents folder on your local computer.

  • NFS – “Network File System” is used for Unix and Linux based operating systems (this includes MacOS).
  • CIFS – “Common Internet File System” is used for Windows and MacOS operating systems.
  • Multi-Protocol – Allows for both CIFS and NFS Connections to the same Turbo Volume, providing access for both Windows and Linux Systems simultaneously. If you wish to request a multi-protocol volume, select the “Turbo Research Storage NFS” and set the “Multiprotocol” option to “Yes”

Mounting Your Turbo Volume

Mounting NFS Volumes

For Linux based systems:

To Successfully mount NFS Storage to your machine, you will need:

  1. Static Hostname/IP Address for your client.
  2. Client Hostname must be added to the NFS Client Export List for this volume in the SRS.
  3. NFS Utilities to be Installed

Mounting an NFS Volume:

>> mkdir [MOUNT LOCATION]
>> mount -t nfs [STORAGE VOLUME] [MOUNT LOCATION]

Storage Volume = [NAME].turbo.storage.umich.edu:/[NAME]

Mount Location = Local Directory the NFS Storage is attached to (usually /mnt/[NAME])

A Key Security feature of NFS Storage only allows access to the volume if the requesting system is included in the NFS Client Export list for that volume. The NFS Client Export List is managed on Turbo Itself by the ARC Storage Team, and requests to add/remove clients to the NFS Export list for a Turbo Volume can be managed through the SRS

There are many options for mounting and configuring NFS Storage (version, persistence, autofs, security, etc).  We recommend working with Unit IT Staff to ensure consistency, ease-of-access, security compliance, for accessing your Turbo Storage.

You should note, that If you are using multiple NFS systems to access the same Turbo Volume (i.e. Great Lakes and a Local System), you will need to ensure that the User Groups are synchronized between these systems; otherwise you will not be able to access your files as expected. If you have problems or questions on synchronizing UNIX User Groups, you can contact your Unit IT Support Team for Assistance.

Alternatively, if your Turbo Storage was provisioned as Multi-Protocol and mounted through NFS on one system, you can safely use CIFS on any number of other systems, without issue.

Mounting CIFS Volumes

Using CIFS/SMB Turbo is accessible to any clients on U-M networks (or VPN with campus profile). CIFS connections are secured entirely through their Active Directory (AD) Permissions. So, if you are a valid user in one of the permitted AD user groups, you will be able to mount your Turbo Storage through CIFS.

For Windows:

Directions are for Windows 10

  1. Open the “This PC” Menu from the Windows “File Explorer” or Start Menu
  2. Click on “Computer” in the menu and then select “Map Network Drive”
  3. Select a Drive Letter from the Drop Down List. Choose a letter after the letter H
  4. In the “Folder” Box, enter the CIFS Volume path
    \\[NAME]-win.turbo.storage.umich.edu\[NAME]
  5. Check the “Reconnect at login” box to have your computer remember this drive mapping
  6. Click on the “Connect using a different user name” and use the following settings:
    • User Name: UMROOT\your-uniqname
    • Password: your UMICH Kerberos (Level-1) password
  7. Click the “OK” button.
  8. Click the “Finish” button.

For Mac:

      1. In the Finder, click on the “Go” menu and select “Connect to Server…”
      2. In the “Server Address” field, type the server name and path:
        smb://[NAME]-win.turbo.storage.umich.edu/[NAME]
      3. Click the “Connect” button.
      4. When prompted, enter your uniqname and UMICH Kerberos (Level-1) password.
      5. Click the “Connect” button.

      Groups & Permissions

      UNIX Permissions

      If you are accessing your Turbo storage through NFS, your access to those files/directories will be based on their UNIX Permissions. The Permissions are assigned at the object level by the User and Group Permissions assigned to those files. By default, only the creator of a file, and the users in the group inherited from the parent directory will be able to access a file.

      For more information on UNIX Permissions, feel free to check out this guide, or ask your Unit IT Staff for help!

      Active Directory Permissions

      If you are accessing your Turbo Storage through CIFS, your access to those files/folders will be based on their Active Directory (AD) Group Membership. AD Group membership is owned and managed by Unit IT Staff. If you need to change permissions or access to CIFS/SMB Volume, contact your unit IT team for assistance!

      Unix Groups

      In order to set appropriate permissions, and provide access for multiple users on our clusters, local Unix groups must be created by ARC to properly administer your NFS Turbo volume. Any user addition or deletions to Unix groups on any ARC service must be requested by contacting us at [email protected].

      AD Groups

      In order to set appropriate permissions, and provide access for multiple users in CIFS volume, AD groups must be created by your local IT department to properly administer your CIFS volume.

      Multi-protocol volumes require advanced synchronization between your Unix Groups and AD groups.  Contact your unit IT staff to help build these synchronization, and we will work with them to set it up.

      Any user change requests will need to be directed towards your local IT department.

      Snapshots & Replication

      What is Replication?

      Replication is the process of making (and maintaining) an identical duplicate copy from one storage system to another. In the case of Turbo, replicated copies are sent offsite to provide data integrity, site redundancy, and increased availability for Turbo data in the event of a systems outage.

      What are Snapshots?

      Snapshots are a tool provided by Enterprise Storage Systems, such as Turbo, that provide point-in-time rollback for any files/folders that are changed or deleted. Snapshots catalog changes to files/folders (on a nightly basis) and maintain copies of any files that were changed or deleted since the last snapshot. So, if a file is accidentally deleted, it may be recoverable through the previous days’ snapshots.

      • Snapshots can be requested at no additional charge, although the snapshots will count against your available capacity.
      • Your snapshots can be found in /nfs/turbo/<volume_name>/.snapshot/
      • Snapshots are available with the following options:
        • Daily snapshot, retained for 7 days (default, if selected)
        • Daily snapshot, retained for 1 day
        • Daily snapshot, retained for 3 days
        • Daily snapshot, retained for 7 days, and 1 bi-weekly snapshot
      • Snapshots are taken on a nightly basis, so if a file is created and deleted on the same day, it may not be recoverable.
      • Note: If a large volume of files are changed or deleted in a short period of time; the snapshots may consume a considerable amount of one’s allocated capacity.
      • If you suspect snapshots are taking up too much space; please contact [email protected] and we would be happy to help resolve the issue

      Recovering Lost Files

      Turbo volumes that are configured with snapshots will save previous versions of files.  Only files which have been snap-shotted overnight are recoverable.  Files that are lost on the same day they were created may not be recoverable.

      From Linux based clients: To recover files lost from your directory, navigate to the .snapshot directory at the root of your share.

      $ cd /nfs/turbo/flux-admin/.snapshot
      $ ls -1
      daily_2015-08-24-_23-30
      daily_2015-08-25-_23-30
      daily_2015-08-26-_23-30
      daily_2015-08-27-_23-30
      daily_2015-08-28-_23-30
      daily_2015-08-29-_23-30
      daily_2015-08-30-_23-30

      You can navigate to the snapshot directories and copy files back to your file share.
      Note: The .snapshot directory may not be visible until you try enter it with cd.

      From Windows clients: You can recover lost files from snapshots natively:

      • Open the directory that the deleted file was held in.
      • Right click in the directory that the file or folder was stored and select “Properties”.
      • Click on the “Previous Versions” tab when the Properties window opens.
      • A list of snapshots will be displayed.
      • Select the snapshot from which you wish to restore data.
      • In the new window, locate the file(s) you wish to restore.
      • Simply drag the file(s) or folder to their correct locations

      Sensitive Data

      Turbo Storage (along with the Armis2 Research Cluster) are certified for storing Sensitive and Restricted Data. With full data encryption in-transit and at-rest; along with strict access-controls Turbo Storage provides the full suite of technical protections for sensitive data.  This includes HIPAA, FERPA, Export Controlled, and most other regulated data types. All sensitive data must comply with our Data Security policies.

      If you have specific questions or need further details with regard to Turbo’s data protection capabilities, please check out our Data Security Policies, or feel free to contact us at [email protected].

      Modifying a Volume

      How do I make changes to a volume?

      • Go to the ITS Service Request System (SRS) portal.
      • Select either Turbo NFS or Turbo CIFS, depending the type of volume you have.
      • Select “Modify” and on the following page choose the Turbo volume you wish to modify
      • Near the bottom of the page, select “Edit.”
      • Enter in any changes you wish to make.
      • Near the bottom of the page, select “Save” when you are done.
      • The changes will be sent to the ARC Storage team so they can process the request. This can take up to a business day to complete. They will update you as soon as the changes have taken effect.

      Transferring Data

      Globus

      Note: Be aware that if you have a Multi-protocol or NFS Turbo volume that is mounted on Great Lakes, there is no need to set up a separate Globus endpoint for it.
      1. Install Globus Connect Personal. Select the OS you are using and follow the documentation to install Globus Connect Personal, and setup a local Globus endpoint.
      2. Go to https://www.globus.org/ and click on “Login” in the upper right corner.
      3. Select “University of Michigan” as your organization in the drop-down field and select “Continue”
      4. This will prompt you for your uniqname and Level-1 password. Enter these and select “Login”, then authorize with Duo two-factor authentication
      5. You should be redirected to the “File Manager” tab on the left side of the screen
      6. Select the “Collection” field and type your local Globus endpoint.
      7. On the right side of the screen, select “Transfer or Sync to…”.
      8. Select the new black “Collection” field and type in the Great Lakes Globus endpoint. The endpoint is umich#greatlakes
      9. By default, you will see into your Home directory. You can change your path under the “Path” field.
      10. When you’ve selected the files you wish to transfer, select the “Start” button near the bottom of the screen. A notification for the transfer will appear in the “Activity” tab on the left side of the screen.

      Cyberduck

      1. Cyberduck can be used on Mac or Windows
      2. Open Cyberduck and click the Open connection button.
      3. Set the “Server:” to: be greatlakes-xfer.arc-ts.umich.edu
      4. Set your Username: to be your uniqname.
      5. Enter your Kerberos password.
      6. Click connect.
      7. Drag and drop files between the two systems. Click the Disconnect button when completed.

      WinSCP

      1. Connecting to a remote system
      2. Select the protocol you would like to use (recommend sftp or scp).
      3. In the host name field, type the hostname you would like to connect to.
      4. ex: greatlakes.arc-ts.umich.edu
      5. The port number will auto-populate based on the protocol you set.
      6. In the user name field, type your uniqname.
      7. In the password field, type your U-M Kerberos Level-1 password.
      8. Click the “Login” button.
      9. If “Continue connecting to an unknown server and add its host key to a cache?” pops up, click “Yes”.
      10. A Duo prompt will appear. Select your preferred method and authenticate with Duo.
      11. Transferring Files
      12. The left panel of winSCP will be your local machine, and the right panel of winSCP will be the remote machine you’ve connected to.
      13. On your local machine, navigate to the file/directory you wish to transfer to the remote machine.
      14. Drag the file/directory you wish to transfer to the remote machine’s panel. This will initiate the transfer.

      SCP (via the command line)

      SCP uses that basic syntax of:
      > scp [SOURCE LOCATION] [DESTINATION]

      To copy a local file to a remote destination:
      > scp /path/to/file [email protected]:/path/to/destination

      To copy a local directory to a remote destination:
      > scp -r /path/to/directory  [email protected]:/path/to/destination

      To copy a remote file to local machine:
      > scp [email protected]:/path/to/file /path/to/destination

      To copy a remote directory to local machine:
      > scp -r [email protected]:/path/to/directory /path/to/destination

      SFTP (via command line)

      Connect to a remote server with SFTP:
      > sftp uniqname@hostname
      > sftp [email protected]

      Transfer local FILE/DIR to a remote system:
      > get file /remote-directory

      Transfer remote FILE/DIR to local machine:
      > get /remote-directory

      Transfer multiple local FILES/DIRS to a remote system:
      > mput files /remote-directory

      Transfer multiple remote FILES/DIRS to local machine:
      > mget /remote-directory

      Storage Policies

      Maintaining the overall stability of the system is paramount to us. System availability is based on our best efforts. We are staffed to provide support during normal business hours. We try very hard to provide support as broadly as possible, but cannot guarantee support on a 24 hour per day basis. Additionally, we perform system maintenance on a periodic basis, driven by the availability of software updates, staffing availability, and input from the user community. We do our best to schedule around your needs, but there will be times when the system is unavailable. For scheduled outages, we will announce them at least one month in advance on the ARC home page; for unscheduled outages we will announce them as quickly as we can with as much detail as we have on that same page. You can also track ARC at Twitter name umichARC.

      Tags: 
      Advanced Research Computing
      Turbo Storage
      Last Updated: 
      Monday, November 7, 2022 - 00:00