Noting that funding for science has run dry for many researchers, [Gaurav] has built a supercomputer from 200 Playstation 3 consoles housed and chilled inside an old refrigerated shipping trailer. His mission at UMass Dartmouth from the National Science Foundation is simulating black hole collisions with an eye on learning something about gravitational waves.

Dr. [Gaurav Khanna] is no stranger to using PS3 supercomputers to do meaningful science. Seven years ago he proposed a 16-PS3 supercomputer running Linux and managed to convince Sony to donate four consoles. The university kicked in funding for another 8 and [Gaurav] ponied up for the last four out of his own pocket. He dubbed it the “PS3 Gravity Grid” and received international attention for the cluster. For equivalent performance, it cost him only 10% the price of a real supercomputer. This led to published papers on both hacked supercomputers and gravity waves. But that rig is looking a little old today. Enter the Air Force.

Dr. [Khanna] was not the only one using PS3s to crunch data – back in 2010 the US Air Force built the “Condor Cluster” of 1,760 PS3s to perform radar imaging of entire cities and do neuromorphic AI research. With their hardware now expired, the Air Force donated 200 of the PS3s to [Gaurav] for his new build. Now that he has wired them up, the Air Force is donating another 220 for a not-snicker-proofed total of 420.

For those sceptical that the now 8-year-old hardware is still cost-effective, even with free consoles it is marginal. RAM is an issue and modern graphics cards are each equivalent to 20 PS3s. Ever the popular target these days, Sony has the PS4 OS locked down from the get go – thanks Sony. The next cluster planned will be with PCs and graphics cards. For now, [Gaurav] has plenty of calculations that need crunching and a queue of colleagues have formed behind him.

64 Comments

  1. PS3s and Graphic Cards to build supercomputers? It just blows me away that AI may come from hardware advances brought about by the “need” for more realistic FPSs!

  2. Supercomputers often have the same embarrassingly parallel problems to solve as rendering millions of pixels on a screen. Any really parallel tasks benefit from this common architecture. The common video card is not that different from a super computer of yesteryear with over a hundred different cores dedicated to single tasks that can be done in parallel.

  3. Generic Human

    Today’s GPUs are actually almost ideal for many supercomputing problems. As greater demand for realism in games led to more complex and powerful shader programs, it became possible to do general purpose computing on graphics cards (known as GPGPU computing). Further, since graphics rendering is an ’embarrassingly parallel’ problem, GPUs are designed to do many calculations at once, just like supercomputers. There are even ‘graphics cards’ that don’t actually have a video connector on them (ex. Nvidia Tesla), and are designed specifically for scientific computation.

  4. Pinky's Brain

    Ideal … except if you’re the one who has to do the programming.

  5. They only work well if your problem is big enough and cache-friendly. GPUs don’t make sense for the low FLOP/byte cases where memory bandwidth is the limiting factor.

  6. I’m told they also become problematic when either the data bandwidth between units goes beyond the scale of the interconnecting bus, or the problem requires results from calculations being done at the same time.
    The first becomes the problem of the interconnect and backbone designer of the hardware, and memory management of the programmer
    The second issue can only be solved by carefully structuring the problem to be solved, or going into different architectures (such as shared memory, shortening instruction execution steps, etc) because many of the techniques used to accelerate mainstream processors (path prediction, fancy instruction extensions and the like)
    If you want to discover how this can affect hardware, you can google Beowulf Clusters to learn about node-style computing and look at something like the Cray-MP1 which did (iirc) 64-bit multiply add in 7 clock cycles; and to make things even more awesome, the result starts coming out into registers part way through and you could feed it into a subsequent operation.
    Or look at CSIRAC (which has an emulator and played the first digital sound / chiptune, the first interpreted language and is the only complete – albeit non functional – first generation computer) which could load essentially it’s entire heard drive direct to registers.
    TL;DR: It’s also about the width and depth of the problem. And CSIRAC rocks

  7. i know – it used to be that supercomputer technology was built to satisfy the needs of science and the military industrial complex. Now the hardware is driven by unwashed nerds living in their mother’s basements. Progress!!

  8. 420 in a reifer goin’ down the subatomac road.

  9. “the Air Force is donating another 220 for a not-snicker-proofed total of 420”
    I’d just like to remind everyone that the RSX Reality Synthesizer is based on an nvidia chip.
    >blaze it

  10. Pinky's Brain

    Which is going to be doing bugger all, it’s all about the Cell.

  11. Why is this on hackaday like it’s new? This isn’t a hack, it isn’t new. We ran racks and racks of PS3s at our University back when I was at university about 8 years ago. It was a cheap compliment to the existing supercomputer.

  12. Are you by any chance from Dr. [Khanna]’s Uni?

  13. anything else you’d like to fill us in on? this is new to me, and i consider it somewhat of a hack, so i think it’s fair to assume you also have some other ‘old’ news you need to share with the rest of us, you know, so everyone else can have your viewpoint of what is new and what isn’t–just so we don’t run into this same bothersome problem in the future..
    the tip line is open to anyone!

  14. he needs some FPGA backplanes +pcie graphics cards…..
    something that can be built for $20 USD for the pcb not inc. the fpga
    I recently picked up some Virtex5+inbuit PSU+SATA interfaces for $20usd each…. working….

  15. please define the word “need” and why it is the appropriate word for this context

  16. Pinky's Brain

    Using slightly older PCIE signalling speeds you can probably do it with slower FPGAs too if you are going for cheap … but if you are buying NVIDIA Teslas going that cheap doesn’t really make sense.
    Ray Bittner at Microsoft already did it BTW.

  17. Here’s a tip line for you: PS3s had an “Other OS” option when they were shipped. Almost literally the first thing someone did with it was install MPICH on it and proclaim they created a “cheap supercomputer”.
    I mean shit the process of clustering PS3s has it’s own damn Wikipedia page: http://en.wikipedia.org/wiki/PlayStation_3_cluster
    For more nerd porn just type PS3 cluster into Google Image search and you can find all manner of PS3 “supercomputers” in various sizes.

  18. A tip 7 years older than the project in the article is your answer?
    I believe the point being made was, submit tips of newer projects if this is too boring and old for you.
    Speaking of that wiki link though… what is the 2nd PS3 project listed there? “In Summer 2007, Gaurav Khanna … PS3 Gravity Grid.”
    He spurred the adoption of PS3 supercomputing, published a paper on it.
    If using 420 shockingly-obsolete machines to discover new physics isn’t hack enough for you, yes, please, submit all the new things that are amazing hacks to you so we can cover those instead.

  19. Not new? He just finished it all of last week, and the 2nd truckload of PS3s from the airforce hasn’t even shown up yet. His previous project of only 16 is the world’s most famous example of PS3 supercomputing clusters, something he published a paper on afterwards so that others could do the same.
    Not a hack? Hrm. So, the Playstation 3 gaming console is being used for its original purpose of being clustered to hundreds of other units as a supercomputer investigating how Black Fucking Holes Collide, whilst inside a container that was designed to chill said hundreds of gaming consoles after serving many years chilling milk during transportation?
    How mind-blowingly amazing is the alternate reality you live in where this to you is not a hack? And by all means, submit any and all things you *do* think are hacks from this universe, ’cause, I’ve got a lot more writing to do.
    The real story to me wasn’t “this is new and exciting”, because as you said, it’s not. The story is “This nearly-decade-old machine is being donated by the hundreds and still doing something useful for our species that can’t manage to get funding otherwise.”

  20. Wow, mad?
    And I hate to be that guy, but what he’s paying in simple electric bills for the PS3s and cooling, he could be putting towards a much, much more cost efficient system.
    All this “hack” is is someone performing a demo of something that’s already been done before. Multiple times. Cool? Totally. Neat? Absolutely.Interesting price point? Without a doubt. But a hack? not really. If this were a guy with a bunch of video cards then nobody would bat an eye at it. Instead it’s a guy with a bunch of PS3s.

  21. If my jovial, sarcastic tone was not obvious enough, no, I’m lightly amused, not angry.
    I agree with some of what you said. I’m kinda shocked that it broke even, even with free hardware. I know it’s not quite apples-to-apples, but a fairly-related measure of worthwhile computing power is to just observe whatever Bitcoin miners are doing. They’re the ones on the bleeding edge of profitability (with logarithmically-increasing value to doing calculations, something a science simulator isn’t necessarily constrained by).
    Let’s ignore setup and maintenance costs, maybe he’s using student slaves. The power cost of running 200 PS3s compared to only 10 graphics cards has to be massively (almost linearly?) higher. ~0.300KW * $0.10KWh * 24h/day * 200 units = $144 per day in power. $52,560 per year unless I goofed.
    Suppose you had to buy 10 modern graphics cards and support systems for them. I’m not familiar enough with stuff to know what they’d choose, but, I’m pretty sure it wouldn’t be much more than 10% ballpark of that $52k… meaning this rig burns more than it’s replacement value in about a month.
    Yet this is not a dumb man, and by observation he still did it. So, somehow it still made sense to do.
    I wonder if power costs never came into consideration because they’re paid out of a different budget. Bureaucracies can be funny. “Hi, can we have $5,000 to build our own supercomputer?”, “Sorry, no budget.”, “Hey, can we run a cluster of 200 PS3s to do simulations 24/7?”, “How much do they cost?”, “They were free, the Air Force donated them”, “Well, fine, go ahead as long as they’re free.” *whistles innocently*

    As to whether it’s a hack, it might not be an impressive hack to you, but, it’s certainly, unequivocally, using something creatively outside of its original purpose. It’s okay if you don’t like it. We post ~8 stories a day. If you’d like to influence the content we cover, rather than complain in the comments – which does nothing but make writers feel shitty about something they put effort into for you to enjoy for free – just submit the kinds of things you feel *are* hacks we should be covering and some of them will probably get picked up.
    For what it’s worth, the tip came in a few days ago and I sat on it until I realized it’s not just and old hack… it’s… surprisingly old of a hack for someone to still be doing. The PS3 is 8 years old already.

  22. You hit the nail on the head with the power budget stuff. At my university no one would give a shit about power consumption. I could run an entire building full of bitcoin miners and power consumption would be the last of the “budget concerns” someone would notice because it’s simply a different department and budget entirely.
    I’ve even thought of using university power to charge up massive battery packs so I can drive them home each day and use grid tie inverters to sell the power back to the grid during peak demand >:)
    This guy got a bunch of free PS3’s and he has effectively FREE POWER, so ultimately he saved himself 5K on a bunch of GPU’s. It’s a good hack in my book.

  23. It’s just yet another application of the whole “cluster made from commodity hardware” idea, so it’s a bit boring. I guess you don’t remember how Slashdot used to be filled with stories about Beowulf clusters. Heck, back in 2003 NCSA build a cluster out of Playstation 2s (http://web.archive.org/web/20071012105203/http://arrakis.ncsa.uiuc.edu/ps2/cluster.php)! Sony themselves also designed the “GSCube” (http://en.wikipedia.org/wiki/GScube) which ran 16 PS2s in a single cabinet.

  24. It is a hack, and it was pretty new to me, I also didn’t know about the military grid, so I found this pretty interesting.

  25. dominoembedded

    Bore off.

  26. The news part is people are still doing this. The hack, was the cargo trailer for the cooling, plus the ps3 running linux in a massive 420 node cluster. It’s pretty amazing to me.

  27. I’m surprised Sony donated anything at all to any research project given their propensity to not allow anyone who owns said console to actually do anything, you know, useful.

  28. Correction: Not own, possesses.

  29. Well, they were young and foolish back then, still fighting Xbox’s HD-DVD in the race for early-adoption of the next media format. At the time they were still open to PS3s being used for science. It was a hell of a publicity stunt. A couple years later with the Blu-Ray trophy in their case they released a patch killing off Linux builds.

  30. fuzzyfuzzyfungus

    They locked things down increasingly tightly as time passed; but ‘OtherOS’ was originally a Sony-sanctioned feature(albeit one that graciously permitted you to run your OS under a hypervisor that more or less entirely neutered access to the GPU).
    I don’t know if it was some plan like the PS2 linux thing(were some tax status classification was at issue) or if the idea was that IBM and Sony wanted ‘Cell’ to take off, and so using the (crazy cheap by dev kit standards) PS3 as a widely accessible introduction to cell programming was the strategy to do that. Once that didn’t pan out, they ended up killing ‘OtherOS’ entirely; but in the beginning they actually had their own tweak of YellowDog Linux(because the PS3 was functionally a PPC system, basically a RAM-starved G5 with SPEs, and you could get the SDK stuff from IBM to fiddle with the SPEs.
    Had Cell actually gone anywhere they likely would have continued the program, at least in some form(maybe ‘dev only’ units, if they were so concerned about hackers and piracy); but using the PS3’s mass market hardware to sell people on SPE-based processing was the plan, originally.

  31. The Playstation 2 tax dodge was that for a time in the UK, every PS2 came with a bundled copy of Yabasic. The Linux kit came later, and was something you had to buy separately so wouldn’t have qualified for the tax break anyway. The Linux kit and OtherOS were more akin to the PS1 Net Yaroze program.

  32. I have a PS One and Two, how many black holes can I simulate with that? Will it help if I use an old Wii also?

  33. Well the PS1 and PS2 were much older and used even more antiquated hardware but… running the numbers it seems the PS2 can’t handle black holes. It’ll calculate neutron stars for you. PS1s, only red giants, sorry.
    The Wii can be used in its original format, simulating beach volleyball physics from your couch.

  34. This type of computing cluster works great when there are floating point calculations involved. For things like prime number factorization and pretty much anything what requires searching through a large set of data, this approach isn’t very useful. Still cool though.

  35. If those are the original, four USB port PS3’s with full hardware PS2 support, I’d like a freebie from the USAF. Then I can trade or sell off my last of the fat ones PS2 – the model when Sony finally figured out plastic was a bad choice of material for the laser sled.

  36. I ran ‘other OS’ on my PS3 for a little while. The newer Sony firmware gets rid of it, so you can’t boot linux. The problem is that you also need software from IBM to get the best use out of the Cell Broadband engine (what they call the processor in the PS3). It’s a power processor with 7 special processing engines (but in the PS3’s they usually had 1-2 of these either wired off or disabled at the factory, or the fab didn’t quite turn out right). You needed IBM’s software to turn on the SPE’s or you had a Power processor running about as fast as a pentium 4 at about 1.8 GHz. The other problem with the PS3 is the lack of ram. It comes with 256 MB of ram. And its Direct Rambus DRAM. And there are no busses on the PS3 to accept more (and no pins on chips to wire in more of your own). The CellBE CPU looked promising when I bought my PS3, but its really a gaming console first. You can make it work like a supercomputer, but its like knitting a sweater from barbed wire: you can do it, but its painful to do, and hurts even after you are done.

  37. fuzzyfuzzyfungus

    For a time, at least, IBM, or one of their minions, sold a ‘proper’ cell-based computer(eg. lots of DIMM slots for RAM expansion, enterprise-worthy boot options, rack mountable, that sort of thing); but as I recall it cost an absolute fortune compared to the PS3 or an x86 stuffed to the gills with GPUs. I don’t think the product ended up going much of anywhere.
    There was also Toshiba’s “SpursEngine” product, which ditched the PPC core and put some SPEs(four, I think?) on a PCIe card(or, in the case of the ~1 laptop that shipped with it, on the PCIe bus; but physically on the motherboard). More promising in theory, since you could use an x86 and its giant pool of dirt cheap RAM, with as many ‘SpursEngine” cards as the motherboard could handle; but never achieved any real traction, and real-world use was limited to a couple of relatively uninteresting video encode/decode accelerations that were overtaken by fixed-function processors not long thereafter.
    As an enthusiast of curious and interesting approaches in hardware design, the ‘Cell’ story is a bit sad; but it’s hard to actually argue that the Cell is anything but a lost cause now that general-purpose GPU computation is on the table and it’s downright difficult to buy a GPU with less RAM than a PS3, and downright trivial to stuff a boring x86 board full of GPUs.

  38. That’s interesting, ISTR Toshiba were developing their own rival to Cell at one stage for incorporation into all their products, some sort of VLIW CPU which had multiple execution units.

  39. IBM actually found the Cell unsuitable for scientific computing. A big part of it was that it only supported single-precision floats. They did develop a successor that also supported doubles, but I doubt it got any customers, and the whole thing was buried pretty quickly.
    AFAIK, Toshiba did use SpursEngine chips with 1-4 SPEs in their early digital TVs and Blu-Ray players, but that was also a pretty short-lived thing.

  40. The spu’s in the ps3 does support doubles*, but it a bit slow mul/madd (5clk stall + 6clk latency instead of 1clk + 6clk latency for floats if I recall correctly, a few years ago I could recite spu op latency in the sleep almost). The follow up model had no stall on the double madd, but it didn’t get anywhere for various reasons.
    *strangely enough there is no double cmp, so if you need ieee double complaint cmp you need to do some quite expensive software emulation.

  41. not really new…
    Journal of Computational Chemistry Volume 30, Issue 2, pages 268–274, 30 January 2009
    Accelerating molecular dynamic simulation on the cell processor and Playstation 3
    This is only the date of publication. You can imagine the testing has been years before…

  42. Peter Kozlowski

    Agreed to those who mentioned that this isn’t new. Why is hackaday promoting stuff like this? Are they that desperate for articles?. Alot of universities and research centers were doing that many years ago.

  43. Blue Footed Booby

    Where on earth did you get the idea that part of Hackaday’s mission is to post exclusively things that have literally never been done before?

  44. I found it an interesting and valuable article, a good read.

  45. One wonders why you, with your vast knowledge in the field, would click on the article. Was the headline deceptive in some way?

  46. just think, without articles like this, he’d have nothing to say

  47. and this is why I don’t post stuff I do here

  48. I didnt know about clustering playstations. and i enjoyed all the comments that wern’t like yours, its been interesting. I built my first hybrid mini over 10 years ago so spose they shouldnt post that, and im quite sure clocks, routers and wire spools were invented a while ago too.

  49. This researcher must be like misusing research funds or something (yes i do realize alot of the equipment was borrowed). In no way is this practical to do/use There’s alot of more practical setups. This seems like the kind of idea he went to the department and was like hey let me tie some ps3’s together its good publicity! Game systems + Research= kotaku news headline.
    Tieing some ps3’s together does not make a supercomputer this screams publicity baiting.

  50. by borrow i meant donate. Still in terms of practicality and feasibility if I was his boss I’d seriously question his use of funding and time. Maybe he hopes his superiors are technology ignorant and he can whitewash this use of ps3’s in lieu of real practical hardware setups

  51. And yet “Alot of universities and research centers were doing that many years ago”. How did they not know of your concerns back then?

  52. That was back when the ps 3 was relatively new and you “could” argue it was cheaper tech wise
    nowadays? it seems more a gimmicky publicity sideshow when there are more more practical alternatives. This guy is trolling for attention on an idea that others already did several years ago. There are better solutions now. Makes interesting read “wow guy using ps3’s as supercomputer” but its false.
    Your also implying university superiors are inherently tech savy. It’s very easy to word something and justify its expense especially if the boss knows nothing about technology. There are universities that get roped into buying many raspberry pi’s which are slow as hell and not practical but it makes great publicity

  53. “This guy is trolling for attention on an idea that others already did several years ago. ”
    This guy basically wrote the book on PS3 clusters for research and is the most famous example of such. I don’t think he needs your approval or praise.
    “Makes interesting read “wow guy using ps3’s as supercomputer” but its false.”
    I’m confused. You are now accusing him of *not* creating a supercomputer out of PS3s and think the photos and New York Times article is a hoax, or, you think that now that he’s built this cluster he is secretly not going to be doing any calculations with it because all he cares about is fame?
    Come back down to Earth, the air is thin up there.

  54. Ok, you’ve got me on assuming competence of any administrator, although you’ll note I didn’t include the Air Force in my post. “Would Da Government waste our money?!”
    That being said, what is your solution? There are
    many, you said. You didn’t like that they turned a pile of junk into a supercomputer thirty times the size of their last one. Not a hack, apparently, because they had experience doing it.

  55. great logic you have there: he’s misusing funds and he wants publicity for doing it
    yes indeed someone who is breaking the law is gonna invite the press to photograph and document the whole thing

  56. Give the correct credit HaD! This is from the University of Massachusetts Dartmouth, not the implied Ivy league Dartmouth College.

  57. You’re right, that was ambiguous.
    Since the new website format only shows a couple sentences on the front page, by which any reader has to decide whether to read more, I try to be as brief as possible. If I added “University of Massachusetts” it would have bumped off “simulating black holes” which is context-critical.
    The very first link describes which university he works at in more detail. And, Dr. Khanna previewed the article and didn’t comment that it was unclear. But, you’re right, and I’m a stickler for details like this. I’ll add in UMass.

  58. Is that Mr. Bean?

  59. Too bad Black holes do not exist. They were created to fill holes in the Math model of cosmology.

  60. Jeff, I think I speak for many of us here when I say that you don’t have a fucking clue what you’re talking about.

  61. eloquently put Sir!!

  62. I believe part of the appeal of the PS3 was the POWER architecture and the many SPEs, combined with a unique programming environment. The PS4 on the other hand is ultimately a conventional x86 with AMD parts, which probably isn’t very appealing for these supercomputing projects when cheaper commodity hardware can do the same job.

  63. (This reply is in support)
    I could buy an nVidia Tesla 40c for about $3.3k; round it up to $4k for the box and control system
    2880 CUDA Cores
    12GB GDDR5 RAM, 288 GB/s Memory Bandwidth
    4.29 TFLOPs Singe Precision Performance
    1.43 TFLOPs Double Precision Performance
    Or ten PS4 with (each)
    8GB GDDR5 RAM, 176 GB/s GPU-> memory bandwidth
    – shared memory, 4.5GB usually available
    2 nodes of 4 CPU cores (split L2 cache)
    – only 6 cores for client applications
    18 core GPU, 1152 shader units
    1.2 TFOPs Single precision GPU
    102.4 GFLOPs Single precision CPU (assuming 100% utilization)
    So for roughly $4k you get a theoretically usable
    45 GB ram
    60 CPU cores, 180 GPU cores
    13 TFLOPs Single precision
    Excellent instruction set optimization on CPU and CPUGPU bandwidth
    Significant difficulty in the data bandwidth between nodes
    And zero ability to use any of it because many academic tools use CUDA and no academic department could afford the SDK. There is no linux on PS4 unless an engineer has stuffed up big time or we start decapping chips.

  64. I want to point out one cool gpu card: 7970 Ghz edition. 4 TFLOPs single precision and 1 TFLOPs double precision. It is from 2012, so you can buy them on ebay for $160-$170. Funny enough next generation has worse double precision performance, so this card really makes sense. Programmable through OpenCL. So to reach 13 TFLOPs, you need only 4 of them, which you can stuff into a regular PC. Total cost – $1200 max. Those are truly wonderful times we are living in. Supercomputers were reaching teraflops only somewhere between 1995-1999…

Leave a Comment

Your email address will not be published. Required fields are marked *