2/7/24: There are no real updates except to say this issue is intermittent. The storage vendor has arranged for additional hardware to be shipped immediately to CCR to help mitigate this issue. We will also be getting an expansion to this storage system in April which will increase not only our storage capacity but also our throughput.
We are aware of overall system slowness that you may notice as slow logins, slow filesystem actions like listing directories, saving files, deleting files, and many other tasks you're trying to complete on our systems. This is related to high load on the Vast storage which serves our /user, /projects, and /vscratch filesystems. This slowness is directly related to latency issues on the storage and it comes and goes depending on the jobs running. The jobs aren't doing anything wrong but we're evaluating them to see if there is a better way to run them. The storage vendor is monitoring and reviewing the system logs in real time. We're working with them to mitigate the issues. Thanks for your patience!
Dori Sajdak
2/7/24: There are no real updates except to say this issue is intermittent. The storage vendor has arranged for additional hardware to be shipped immediately to CCR to help mitigate this issue. We will also be getting an expansion to this storage system in April which will increase not only our storage capacity but also our throughput.
We are aware of overall system slowness that you may notice as slow logins, slow filesystem actions like listing directories, saving files, deleting files, and many other tasks you're trying to complete on our systems. This is related to high load on the Vast storage which serves our /user, /projects, and /vscratch filesystems. This slowness is directly related to latency issues on the storage and it comes and goes depending on the jobs running. The jobs aren't doing anything wrong but we're evaluating them to see if there is a better way to run them. The storage vendor is monitoring and reviewing the system logs in real time. We're working with them to mitigate the issues. Thanks for your patience!