9/1/17: RESOLVED: Isilon directories intermittently unavailable

We’re aware of an issue with the Isilon storage system that supports your home & project directories and the /util file system.  Though the issue has been identified, we’re waiting on the vendor to supply a fix.  We should hear from them within the hour.  In the meantime, you’ll experience intermittent problems connecting to these directories.  Some nodes that jobs are running on will be fine and the jobs will complete without an issue.  Some nodes may have a problem and your jobs may fail.  We’ve paused all new jobs from starting so as not to cause any additional issues for your work.  Once the problem is resolved with the storage, it will take us a bit of time to clean up any problem nodes but we’ll take care of it as quickly as possible.


Please do not panic that any data is lost.  This is only an issue of one server in a redundant group of servers not behaving properly.