EWS Linux Home Directory Update - October 19, 2013

The issues we are currently struggling with on the Linux home directory server are not the same as earlier.  Previously, restarting the server would restore performance for some period of time.  In addition, we had learned how to identify signs that the server was building up to an issue and were pre-emptively restarting it to head those issues off.

The current issue differs in two significant ways.  First, the server appears to go from ok to unresponsive with no notice.  Second, restarting generally does not resolve the issue.  When the problems do start, we have identified on the storage area network a flood of attempts to write 0 byte files.  We are still working to identify the source of these requests.

In addition to actively working on trying to identify the source of the problematic traffic, we are also working to implement traffic shaping targeted at the problematic NFS traffic.  Those changes are being tested and we hope to implement them before the end of this weekend.