Much like an overstuffed filing cabinet, our shared project directories may become difficult to sort through and we may put off cleanup efforts when they become too time consuming or overwhelming.  CCR's quotas limit us in total used space but also total number of files per user and per group, to protect our systems from errant file creations that may crash the storage.  Sometimes it's difficult for us to tell where all the quota is being used, especially when we have a large group sharing one project directory.


CCR is now offering research groups the ability to access detailed information about their data to better sort through and make decisions on what to keep, what to backup up outside of CCR, and what to delete.


PIs or faculty group managers may request a 30 day allocation to the Starfish storage portal.  To do so, request an allocation in ColdFront (https://coldfront.ccr.buffalo.edu) for the 'Starfish' software resource.  The PI is automatically added to the allocation.  If the PI will not be the one doing the data inspection and tagging, please add only one very trusted project member to the allocation.  This person will have access to view all files in the group's shared project directory.  It may be that your group has set the system permissions such that everyone in the group can already read all files; this is CCR's default but many groups change their permissions.  In this case, the permissions in Starfish would be the same.  However, the user with Starfish access will be able to tag files and directories for deletion so please be very careful with this!


If you need multiple group members to make decisions on tagging data, please do NOT share your account information with them.  Request additional access through CCR help.


Once you're done with reviewing and tagging your data, please submit a ticket to CCR Help to let us know it's ok to run the delete process.  At the end of your 30 day allocation, access to Starfish will be removed to free up the license for other groups.



Using Starfish


Once the group has been given access to Starfish the PI will be able to login using their CCR username, password, and one time token.  If you're unsure how to login with CCR's two factor authentication, please see this super short video or this detailed article


https://starfish.ccr.buffalo.edu

As with all other CCR resources, this is only available when on the UB & RPCI networks or their VPNs.




You will be considered a Starfish Zone Admin and have access to detailed usage information for your group's shared project directory.  At the top right, under the Help menu, there is a link to the Zone Admin User Guide.  This contains all the information you need to understand how Starfish works, how it calculates usage (for example, usage is calculated in TiB not TB), and how to tag your data for deletion or other classification.


You may need to be logged into Starfish to view these.  They are also accessible from the 'Help' menu:

Zone Admin Quick Reference Guide

Zone Admin Guide




Visualizing the data by age, size, and other options


Starfish does a great job of showing us our data in various different ways.  This shows the directories by logical size and displays by last access time:



There are many variations on this.  Please refer to the documentation for more information.




Downloading Reports


You are able to download information about your directories into CSV format.  There are a variety of data points you can include and options you can set.  Please note if your group's project directory has millions of files or thousands of subdirectories, you should be cautious about what information you include in this CSV report.  If it's too large you will not be able to open it in Excel.  We suggest creating more general reports for the overall project directory usage and only include one or two levels of subdirectories.  For more detailed information, go into the subdirectory and create more detailed reports for those separately.




Hints


The Starfish GUI offers alot of information and capabilities!  It's hard to keep them all straight and time consuming to learn it all.  Their "Hints" feature is very useful for basic information about each of the icons, tools, and sections.  Click on the "Hints" button at the top right to see all the available hints.  Then hover over the question mark icons to get more info:





Tagging Files & Directories


We provide "action" tags to mark files and directories for deletion in 30 or 60 days and also a "classification" tag to mark a file or directory for off-site backup (archive).  If you mark files or directories with action tags, nothing happens to the data until you tell CCR to move forward with the deletion process. 



Right click on a file or directory to view the different types of Tags and Actions you can take on them:


CCR does not have any archival storage.  However, groups may wish to go through their data, identify files to backup off of CCR (i.e. archive) and then generate a report for their group members to review those marked for archive.  Users would do this on their own outside of Starfish but this classification process is a nice way for groups to periodically review whether old and untouched data needs to remain taking up quota on CCR's systems.  


On the left hand panel under 'Tag Search' you can show just those things marked under the different tags.





All done?

Once you're done with reviewing and tagging your data, please submit a ticket to CCR Help to let us know it's ok to run the delete process.  At the end of your 30 day allocation, access to Starfish will be removed to free up the license for other groups.