image

Amazon S3 was not built on spare capacity

At least, not according to Jeff Barr, Amazon’s Web Services evangelist. I was reminded of this when reading Om Malik’s post on the recent S3 outage, in which he quotes Antonio Rodrigez who asks:

… if AWS is using Amazon.com’s excess capacity, why has S3 been down for most of the day, rendering most of the profile images and other assets of Web 2.0 tapestry completely inaccessible while at the same time I can’t manage to find even a single 404 on Amazon.com? Wouldn’t they be using the same infrastructure for their store that they sell to the rest of us?

I asked Barr a question along similar lines at Qcon London in March. My concern was whether the business model (S3 is cheap) would break once Amazon had to invest in new servers purely to support S3. This is what he told me:

It’s a common misconception that we launched this simply because we had servers sitting around and we wanted to find something for them to do. It’s always been the case that we launched this specifically because we wanted to bring something of value to developers … Our community is about 330,000 developers now.

I presume that there is nevertheless considerable synergy between S3 and Amazon’s own need for distributed storage, and that there is shared software, data centres, and so on. With hindsight I should have pressed Barr further on what is shared and what is distinct. But it isn’t, apparently, excess capacity. I consider this an advantage.

Update: If Barr wants to correct this misconception he could start by clarifying the wording on the S3 home page:

…it gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

One thought on “Amazon S3 was not built on spare capacity”

  1. I notice that during the whole S3 outage there was no mention that Amazon was down as well. This leads me to believe that Amazon is not even using their own service themselves (dog fooding). I think when they start using it themselves the reliability will go up a few fold, after all, if Amazon goes down for a few hours they stand to lose a lot of money (and creditability).

Leave a Reply

Your email address will not be published. Required fields are marked *