Unplanned server outages

A hard drive on the engr_storage array had hardware problems that caused it to enter a endless read loop, effectively making the entire array unavailable to all servers that used it until the hard drive was replaced.  Most Engineering virtual servers use the engr_storage array which meant that they also paused until the hard drive was replaced and, in several cases, the virtual servers shut down automatically.  

Engineering IT staff are on site and have replaced the failed hard drive.  The virtual servers that shut down or went into read-only mode are being checked.  All virtual servers should return to service soon.

EWS users who were using svn services were affected by this outage and logins for Windows lab machines could not complete properly for at least an hour.

ENGR-FILE01 file server shut down and was unavailable during the outage.  Some data volumes will be unavailable while they're being checked for data corruption.