This issue has been resolved
The GPFS system including /gpfs/scratch went offline unexpectedly around 3:30pm today. Our SAN administrator is working with IBM to diagnose and fix the issue. We apologize for the inconvenience this will surely cause with running jobs.
As a reminder, all scratch spaces on CCR servers are considered volatile and are not backed up. They are not intended for permanent storage of data which is why there is a 3 week lifespan on /gpfs/scratch. Anything older than 21 days is automatically deleted nightly. See our scratch policies here for more info