Some thoughts on #HPC in the #cloud... (#askolo question)

layout: post published: true Date: 2012-04-13 Tags: hpc askolo opinion

posterous_url: posterous_slug: some-thoughts-on-hpc-in-the-cloud-askolo-ques

As previously mentioned, I've been spending some time lately on Askolo, a directed Q&A website. My first question from another user was "What do you think of the AWS HPC clusters?", and I wrote a fairly lengthy and rambling response which seemed worth saving. (From 12/9/2012: Askolo went into maintenance mode later this year, and the site's future is uncertain... so this may have been a good idea! :) ) I'm pasting it here (slightly modified) so it gets a different audience, and to keep my own copy for later comparison as my opinions evolve.

Q: What do you think of the AWS HPC clusters?

A: Disclaimer on this one: I'm an engineer at R Systems, which provides bare-metal HPC clusters as a service, in competition with Amazon's HPC offering. Also, this got a little long. :)

That said, I think the AWS HPC clusters are really interesting. They're definitely useful for "bursting" loads and certain classes of problems, though they still have a few problems to solve before they can replace a "traditional" cluster. But for a lot of wokloads, it may be exactly what you want.

For HPC, Amazon provides exactly the same thing it provides for everyone else: a big pool of available servers which can be spun up and down at will, with a potentially huge upper limit. You may have seen the article last year about the 30,000 or so core cluster which Cycle Computing spun up on EC2: it made the Top 500 list of supercomputers, and demonstrated that it was possible to run a huge compute resource "in the cloud". 

There are some drawbacks though. Cycle Computing ran an embarrassingly parallel problem, but a lot of research models are very tightly coupled and do a lot of message passing between processes. (For example, big CFD models and mechanical modeling problems.) Most big HPC clusters currently use Infiniband as their interconnect, so node-to-node the network provides up to 54 Gb/s bandwidth and 1 us latency (process-to-process). The latency is especially important: I think most CFD apps I've worked with lately are latency-bound. Right now the best network performance you can get on EC2 is 10Gb ethernet, which doesn't quite provide the same punch. The disadvantage to IB is that its network model is radically different, to the point where the IP Protocol feels a bit bolted-on and can't get all the network performance. Most people program their apps with specialized message-passing libraries (ie MPI) which can take advantage of the IB network directly.

EC2 also doesn't get the same I/O performance you can get on bare metal. This one's a problem for lots of people, including big web sites, and it matters in HPC too. A lot of HPC installations have big parallel filesystems that stripe over many disks, like Lustre. It'd be interesting to see what you could do running Lustre on EC2, but I think using EBS as the backing strorage would make it somewhat painful. Much nicer to use big I/O nodes attached to Infiniband.

But you notice how much specialized hardware we're talking about here? Lots of big I/O nodes, a specialized network where even IP is a second-class citizen... it all makes sense if you do HPC all the time, but if you only need to run for a few months out of the year it can seem like overkill. Especially if you are in fact running embarrassingly parallel models (and really, a whole lot of them are).

I saw a talk at Supercomputing in November where someone compared their own internal cluster to 3-year reserved instances on Amazon, and determined that Amazon would make sense if their internal cluster had yearly utilization of less than about 70%. That's actually pretty impressive! Most "big HPC" out there has utilization of 90+% and can you get us some more please ;-) , but there are a lot of companies with underused 32-node clusters out there which might make more sense in the cloud.

On a site with so many people running startups I hope I can be forgiven a tiny bit of self-promotion: at R Systems we provide clusters as a service with all the bells and whistles and Infiniband included, and help you use and configure them as well. :-) But that doesn't mean I think Amazon sucks: I've seen their service do amazing things, especially for highly-variable workplace or embarrassingly parallel problems. 

What all this does mean that for HPC, you need to think at least a bit about the hardware, which I guess makes even Amazon a bit less cloudy. You need to think about what kind of inter-node communication you need, ans how fast your IO needs to be. If you use GPUs, you need to know what type ans think about PCI performance. And you can't just throw more cores at the problem, because communication bottlenecks will kill you.

So HPC on AWS is very cool, as is HPC on R Systems or other external providers. But profile your application and think about hardware before you spin up 1000 nodes. :-)