Sony has been working on a new client to fix several client-side bugs which have come to light during the last few days. Once that new client is in, we can set the servers back to normal and all should be well, assuming the client addresses the issues we’re facing. For now, we (Stanford) have been trying to do what we can server-side to work around some critical bugs in the 1.3 client to allow for both client uploads and downloads. It’s easy to get uploads or downloads working, but with the current state of 1.3, it’s hard to get both. We have had to rewrite server code to work around these 1.3 bugs and I think we’ve made some progress.
The bugs in the 1.3 code are very subtle and the sort that only comes up once a critical mass of 1.3 clients exist and start to hit the server collectively in incorrect ways. Sony QA didn’t show this up, as one can’t QA 30,000 clients. However, we have discussed in detail with Sony some ideas for helping to prevent this in the client in the future, and this behavior is being removed for the upcoming patch.
I want to address some misconceptions in the comments posted here. There is a problem in the client, and the server changes were made to address this and work around the bug (the server changes did not create the problem, but were done in response to the problem). Also, Sony QA is extremely rigorous, but these sorts of things can’t be found until clients hit critical mass. Finally, we have tried several server code changes and the reasons they didn’t help appear to be that the client had multiple aspects which didn’t behave as expected (eg client bugs) which rendered our changes not as useful as one would have expected at first (forcing us to reconsider what one can do server-side).
Last night, we tried a new strategy where we use the AS to help control the weight of clients going to the WS. This seems to be working as we’re getting lots of uploads right now as well as a steady stream of downloads. We hope that this is a reasonable balance and if not, we will continue to see what we can do from here until the new client comes out.