One of the stanford nets appears to be down — IT is looking into it

We have 4 different server rooms for redundancy and it looks like one of the room's network is down.  All machines on the 171.64.65.XX net (VSPG*.foldingathome.org) are unreachable right now.   [Note that this server room is in the Stanford Computer Science building and it looks like the whole Stanford CS net is down (eg http://www.cs.stanford.edu/ is unresponsive).]  We have notified the network admins and the server admins and they are working on this right now.  It's Saturday, so they're on a reduced staff, but they're on it.  This is something beyond my group's control (much like an electrical outage or a major natural disaster) and we have to wait for the Stanford CS networking gurus to do their magic. 

The good news is that much of FAH is still up.  You can see what parts are up vs down on our serverstat page (https://foldingathome.org/serverstat.html).  However, the bad is is that with many of the key machines unreachable, the other machines are heavily loaded at the moment.  Once the additional server room comes on line, then everything should ease up.

It's very unfortunate timing that this comes shortly after a rough set of weeks with our complete server upgrade, which lead to several machines being down in a rotation for updates.  That lead to servers being hit hard too.  Note that we have been working to improve client/server performance under such high loads.  That development lead to server code changes and a new client binary available that helps uploads to servers when the servers are loaded.  Please see the previous posts here for updates on those clients.