We have been working aggressively to find the root of this problem, investigating all possibilities (server, client, network, etc). We have been debugging the entire FAH system on the PS3 over the last few days (as there could be several causes for what we’re seeing), examining especially how the server is interacting with the clients, what the clients are doing, and how the Stanford network is handling the situation. Several Stanford network engineers have looked into the problem to see if this is a network issue, but that does not appear to be the case. The servers are running extremely well right now. Our investigation so far has found that this issue is due to a problem
in the client, and we have identified a specific issue that’s causing
this problem.
With the completion of this investigation Saturday night, we have given the info of the results of our debugging and our plan on how to fix this to Sony last night, and we are hoping that they can come up with an updated client soon. Unfortunately with the PS3, we cannot update the client ourselves, otherwise we would release a client update ourselves, as we have done in the past as needed. Therefore, over the last few days, we have worked on server side tweaks (the only part we can work on) while Sony can work on the client. Before the client gets updated, we expect that the situation will continue to improve gradually but with the issues we have been seeing.
It looks like there are some misconceptions about the situation, based on the comments posted here. This is not an issue of the FAH servers not being powerful enough — any server network would be showing this same issue with the client issue that’s going on (in fact, the FAH servers serving the PS3 are more powerful than in other parts of FAH, which is operating just fine with far more clients); indeed, the PS3 backend has been spec’d out to handle 1M PS3 clients and we are way under that. Also, this is also not an issue that the PS3’s are too fast. The server load depends on the number of clients and the amount of bytes they send back; in FAH, the PS3’s compute faster, and that is used to do more complex calculations, not to send more bytes, and thus does not create a greater server load.
The client update would fix how the client interacts with the servers
to stop the problem we’re seeing right now, including the issues with
assigns (getting new WU’s), accepts (returning WU’s), and points. Sony is a large company and the development team likely cannot publicly give out ETA’s on when this will be fixed, etc, but it’s important to stress that they are working on this and know this is a very, very critically important update to make, and they are working aggressively to fix it.