Amazon was a great promise for us in terms of high availability. It supplies a distributed system for storage and computing in the cloud that really had been cost effective comparing to the other CDN (Content Distribution Network) solutions around.
So... we placed our bet on Amazon S3 and were satisfied with it's cost and performance. The graceful degradation mechanism our widget had was based on S3 high availability for serving our scripts. From there on, our script logic was taking care of the graceful degradation.
All Amazon web services had a crash in February 08 but their servers immediately returned 500 error which made the pages simply skip the scripts execution. In the last couple of months their performance from around the globe were getting much worse and we started getting complaints from bloggers and publishers that we are slowing them down. A check we did with the keynote monitoring service did show that AMZ are the cause to the problem as well as complaints from other S3 users in the S3 developers forum.
The most annoying thing was that Amazon kept silence and did not publish anything about it.
As Yaron posted in our blog last week, we have changed from S3 to another CDN solution and things have instantly improved.
My believe is that in technical systems breakage happen, it is natural. However, the bigger you are you are expected to have less breakage since you have many more resources to make sure it wont happen. Specifically Amazon that each minute of their downtime is high cost you would expect it to be very reliable. Apparently some users (outside US) are less valuable for them then the American one and they maybe chose to not invest in S3 high availability abroad.
I can understand downtimes of twitter and small start ups with very high traffic. Not pleasant but understandable. I totally did not expect this from AMZ.
I guess S3 lost a lot of customers due to this failure and their silence about it.
It is sometimes better to say "We know there is a problem and working to fix it" then not saying anything.