RESOLVED: 7/27/22: Known issues with Panasas scratch
D
Dori Sajdak
started a topic
over 2 years ago
The system has been stable for the last 5 hours so we are marking this as resolved. Please contact ccr-help if you notice continued issues with using /panasas/scratch
7/29 11:00am: The system has been stable for a few hours and response times have improved. Our support case is still escalated and we continue to monitor the status. We appreciate your patience!
7/29 8:30am:Mitigation procedures are still taking place. Our storage admin is watching the system and reporting all issues to support. Due to the file delete issues they have had to lower the threshold again which reduces the effective speed of the filesystem.
7/28 2:30pm:All volumes fell offline making /panasas/scratch directories unavailable to users and running jobs. We have escalated the request through the vendor.
We are aware of slow response times with Panasas global scratch directories. We have been actively engaged with their support to resolve the issues. They were seeing users delete massive amounts of files at the same time which was causing instability of the storage systems when trying to keep up with these delete requests. To prevent the system from completely crashing, they have had to put in mitigation protocols to slow these delete processes down. However, the side effect of this is a reduction in response times for these directories. We appreciate your patience while these processes run to completion.
Dori Sajdak
The system has been stable for the last 5 hours so we are marking this as resolved. Please contact ccr-help if you notice continued issues with using /panasas/scratch
7/29 11:00am: The system has been stable for a few hours and response times have improved. Our support case is still escalated and we continue to monitor the status. We appreciate your patience!
7/29 8:30am: Mitigation procedures are still taking place. Our storage admin is watching the system and reporting all issues to support. Due to the file delete issues they have had to lower the threshold again which reduces the effective speed of the filesystem.
7/28 2:30pm: All volumes fell offline making /panasas/scratch directories unavailable to users and running jobs. We have escalated the request through the vendor.
We are aware of slow response times with Panasas global scratch directories. We have been actively engaged with their support to resolve the issues. They were seeing users delete massive amounts of files at the same time which was causing instability of the storage systems when trying to keep up with these delete requests. To prevent the system from completely crashing, they have had to put in mitigation protocols to slow these delete processes down. However, the side effect of this is a reduction in response times for these directories. We appreciate your patience while these processes run to completion.