Folding@home Team 56895 and Life with PlayStation

[AlaskaDon]

PS3 server issues, update #5
Some good news. The servers have continued to improve (25-30% failure now, which means that at least 1 out of 2 attempts should work, so only ~2 retries should be needed). We've been implementing and testing lots of different strategies and I think we've found one that works best. We've also been in close contact with Sony and they have some ideas on the client side and are working on revisions there.

We've also drasitically brought down the assigns on vsp06, the server which was assigning large WU's, so to make sure that it's not loaded when those WU's need to come back (which should be about now). So, the bottom line is that the server load is still extremely heavy, but the situation is continuing to improve. Most importantly, the client mods should prevent this from happening in the future.

That's the latest on the upload situation. Hopefully all the non-credited work will catch up eventually. I finished one of those 1500 pointers this morning that I haven't got credit for yet. I have a normal sized unit due to finish in a couple of hours so I'm going to make sure that I'm watching when that finishes to see what happens.

[mystiksuicide]

I have stop folding for the time being since I have like 5 days of uncredited work. It's not about the points but why should I leave my comp and ps3 running if the works is going through.

[powersfoss]

I know I was disappointed when my 1,200,000 frame work unit got lost with their server issues. I'd never seen a PS3 work unit so big and I couldn't wait to see how many points I'd get for it. At least they're well aware of the problem and are working on a fix as quickly as they can. I'm going to leave my PS3s folding and just hope they start counting work units again soon. I'll never catch up with melikefish at this point. I think most of his work are done by overclocked PCs and therefore he's not affected by this. He can sail straight up to the top while we all have to suffer due to Stanford's problem with the PS3s.

I guess the PS3s are just doing too much work for Stanford's folding@home servers to keep up!

[powersfoss]

Here is the latest on their attempts to resolve the issue. My take on this latest update... Sounds like they've done all they can on their (Stanford's) end and are relying on Sony to push a folding@home update out to us. My guess is that would happen on Thursday (since all Sony updates come out on Thursday) at the earliest. <Sigh>

FEBRUARY 03, 2008
PS3 issues, update #6

We have been working aggressively to find the root of this problem, investigating all possibilities (server, client, network, etc). Our investigation has found that this issue is due to a problem in the client, and we have identified a specific issue that's causing this problem. We have been debugging the entire FAH system on the PS3 over the last few days (as there could be several causes for what we're seeing), examining especially how the server is interacting with the clients, what the clients are doing, and how the Stanford network is handling the situation. Several Stanford network engineers have looked into the problem to see if this is a network issue, but that does not appear to be the case. The servers are running extremely well right now. The problem appears to be how the clients are talking to the servers.

With the completion of this investigation Saturday night, we have given the info of the results of our debugging and our plan on how to fix this to Sony last night, and we are hoping that they can come up with an updated client soon. Unfortunately with the PS3, we cannot update the client ourselves, otherwise we would release a client update ourselves, as we have done in the past as needed. Therefore, over the last few days, we have worked on server side tweaks (the only part we can work on) while Sony can work on the client. Before the client gets updated, we expect that the situation will continue to improve gradually but with the issues we have been seeing.

It looks like there are some misconceptions about the situation, based on the comments posted here. This is not an issue of the FAH servers not being powerful enough -- any server network would be showing this same issue with the client issue that's going on (in fact, the FAH servers serving the PS3 are more powerful than in other parts of FAH, which is operating just fine with far more clients); indeed, the PS3 backend has been spec'd out to handle 1M PS3 clients and we are way under that. Also, this is also not an issue that the PS3's are too fast. The server load depends on the number of clients and the amount of bytes they send back; in FAH, the PS3's compute faster, and that is used to do more complex calculations, not to send more bytes, and thus does not create a greater server load.

The client update would fix how the client interacts with the servers to stop the problem we're seeing right now, including the issues with assigns (getting new WU's), accepts (returning WU's), and points. Sony is a large company and the development team likely cannot publicly give out ETA's on when this will be fixed, etc, but it's important to stress that they are working on this and know this is a very, very critically important update to make, and they are working aggressively to fix it.

[Jafo]

Thanks for the update, powersfoss, as I just ran into this problem today.

[AlaskaDon]

Hopefully you won't have lost that big work unit. The client has a que whiich can hold ten units and there is a long deadline on it so it may still upload ok.

[WriteSimply]

I got a 1.4Mb WU in the last 24 hours. When I first discovered it, it was about 30% done and I still have 1 day and 4 hours to go. It's huge but seems to be running on a smaller number of SPEs. I can tell because the PS3's fan doesn't blow hard.


fuad

[melikefish]

Originally Posted by powersfoss

I know I was disappointed when my 1,200,000 frame work unit got lost with their server issues. I'd never seen a PS3 work unit so big and I couldn't wait to see how many points I'd get for it.
...
I guess the PS3s are just doing too much work for Stanford's folding@home servers to keep up!

What was the project number? I thought one could look up the points for any active project from this page: http://fah-web.stanford.edu/psummary.html

I don't see a 1.2MM frame work unit (max of 100 listed); the biggest amount of points for a PS3 work unit is listed at 1500 for projects 4024 and 4025.

That seems like a lot of points, until you consider you can get 250 points for a PS3 WU that is 1/6 the size (such as 4023) of 4024/4025, which will result in basically the same points per day total. So unless their time calculations are off (imo doubtful since all PS3's have the same CPU/GPU hardware), I doubt you would be getting any net points per day gain with these big WUs - probably a chance for less, since if you lose the big WU for any reason you've lost all of that at once (vs in smaller chunks as would be the case in small work units).

Yet another example of why points for PS3 work units should be increased....

[AlaskaDon]

Projects 4024 and 4025 show up as 1,200,000 frames and are worth 1500 points. I completed 4024 on Saturday, but unfortunately it's still in limbo. Both those WU's have 24 day deadlines so the servers should be fixed before the WU's expire. Usual deadlines on PS3 WU's are like 2 days and people are complaining that they are staying in the que long enough to expire before they are uploaded.

[Shin-Ra]

One Million PlayStation 3 Users Participate in Folding@home Research Project

[powersfoss]

My work units seem to be counting much better now so it looks like they've done something to address the issue in the short term with all their server/networking upgrades. Here today's official update from folding@home on the status of the PS3 server issues they've been having:

FEBRUARY 04, 2008
PS issues, update #7

We have a conference call with Sony today to brainstorm a short and medium term plan. We came up with some new ideas to try to help things immediately (i.e. later today if we can get the coding done, tomorrow hopefully otherwise), and also to plan for updates to the client. The call was very productive and the Sony team is very eager to get this fixed as well. I think we have a good idea to fix this, but we'll see when we implement it.

[JTStarkiller]

Man, this has been really frustrating. Not only do my numbers not match up on the stats page (I have 8 according to Folding, and 18 on the web), but I haven't gotten an update on my PS3 for the longest time now. I've been stuck on 8 for at least a month.

I wonder if my work units will accrue eventually, or if I did all that for nothing.

[mystiksuicide]

I have like 6 works uncounted I am going to stop folding for the time being.

[actionhank]

Originally Posted by Shin-Ra

One Million PlayStation 3 Users Participate in Folding@home Research Project

i wonder how much longer before the system becomes self-aware and starts to farm humans for our body heat then calls itself Skynet

[powersfoss]

Originally Posted by actionhank

i wonder how much longer before the system becomes self-aware and starts to farm humans for our body heat then calls itself Skynet

ha ha ha. nice. Yes, we are all unwittingly charging forward, with every completed work unit, towards creating a sentient computer master race to dominate us. lol

[WriteSimply]

Originally Posted by actionhank

i wonder how much longer before the system becomes self-aware and starts to farm humans for our body heat then calls itself Skynet

Terminator series and The Matrix? You're mixing up your movie metaphors!


fuad

[brrneon]

February 05, 2008
PS issues, update #8
We had a long meeting with Sony yesterday to brainstorm fixes that we can do *before* a new client is released. We've come up with a plan, coded it yesterday, and we've started to roll it out today. The result is that clients will wait a bit to get work, but in their waiting, uploads and downloads for those that do get through will go smoothly. The new client will have these waits set up for general cases, but we think we can generate one in an adhoc manner with what we've set up. If all goes well, this fix should solve the issue in about a day or two. If not, we will do more brainstorming until the new client goes out.

Maybe my WU will be counted now.

[powersfoss]

They're still pluggin away at this issue. God speed, Stanford!

FEBRUARY 05, 2008
PS issues, update #9

We've been trying creative several fixes to handle what the servers can do with the existing clients to improve the situation immediately, rather than waiting for a new client could come out. So far, I think they have largely been unsuccessful, but I think we learned more about what's going on. We have a new idea, which will require some more coding, and then we'll roll that out tomorrow.

However, people should see a major points increase due to some work over here, in particular for the large-point WU's.

Do you think they're increasing the point values for PS3 work units? That last comment is perplexing.

[actionhank]

The longer this drags on the more new PS3 owners will be discouraged from folding. Thank god its feb, not exactly peak buying season.

[brrneon]

They have counted mine now and I think it included the ones waiting to be uploaded. 3 WU for my PS3 counted 3,250 total. Those were not the small ones either, atleast two took 1 1/2 days to complete.

It is cool that the WU count more but if it takes more than a day and only get 1000+ points, then that is less than the 950 daily avg that I was getting before when completing 3 WU.

I am glad they are counting now, but I hope the new longer WU get more points. If not then I still will continue to fold it will just take me longer to get into the 40s.