Mobile Version

Latest on


  • Buy Israeli handmade

Save The BlueFin

Blog powered by Typepad

Your email address:

Powered by FeedBlitz

Powered by FeedBurner


« Ops and Scalability stuff | Main | Greening our data centers »

July 18, 2010


Nir Yeffet

Well said. It is a honey trap, also because once you marry to the "cloud services" (for example Amazon's S3, rightscale's scripts, etc.) it is very hard to exit.

BTW, I love Tokyo Cabinet, Maybe I should write something better than Tokyo Tyrant? I can do real wonders with epoll :-)


I agree that you must have sysadmins, however I think ALL developers have some knowledge of administration. You have to know all of your stack bottom up and top to bottom to really understand how the system works, where are the bottlenecks and be able to really scale.

Regarding the cloud, the problem is not the cloud itself, its the platform. Amazon runs most of its own site on its own cloud platform and there are a lot of benefits for that.

Just consider a simple deployment of new version in the cloud vs. physical computers:

1) Start new instances with new code
2) Test them on real production config
3) Using the load balancer route more and more traffic to them. If there are problems you always have your old instances running.
4) If all is ok, move all traffic to new instances and shutdown the old ones.

Physical machines:
1) If you have spare machines or can downsize capacity for a while, install the new code there.
2) Test new code
3) Start moving traffic to new machines / upgraded machines
4) Upgrade the rest of the machines
5) If you have an issue rolling back is a bitch (at least more than shutting down a couple of instances in the cloud).

From a scaling point of view, your architecture shouldn't be dependent on the cloud at all. You should be able to migrate to physical machines (or just use a cloud provider such as GoGrid or Rackspace cloud that provide both cloud and physical hosting).

As you said, you need admins for the cloud, so the skill set stays and knowledge is still required. You just save, at least at first, the 1/4 time spent on networking and hardware.

If a VC tells you that must go to the cloud because its hot, you might want to reconsider taking this VCs money... :-)


Oh, and using Xen, KVM for the raw virtualization or the Eucalyptus project (which Ubuntu's private cloud is based upon) can give you the cloud environment you need.

Today I also heard that Rackspace are opensourcing their cloud tech called OpenStack to the wild. So things are going to be interesting.

Ori Lahav

Eran, I agree on all you have said.
Here is how we solved the deployment/live testing issue.

In outbrain we have few machines that are assigned to "staging" environment. they usually take live traffic and we are able to regulate traffic to them via our load balancer (BTW: HAProxy over Linux - also Open Source). new versions are deployed to the staging Environment and are measured for performance as well as our applications KPIs. once we are happy with it in staging we are releasing to the datacenters one by one so if there is a fault you can shift traffic to an old version datacenter while rolling back the new one.

one thing to have in mind regarding the "flexibility" of the cloud is that usually web applications, in order to get fast response, are keeping as much data as they can in RAM. this makes the cloud less attractive because loosing or giving up machines mean also giving up portion of the data that will hit your performance when you want to recover from it.
Basically if you need your machines to be live 24/356 and serve traffic, there is no reason to keep them in the cloud - it is simply not cost efficient.

As for the VCs, I agree but the problem is that entrepreneurs think the VCs knows better then them and the lack of knowledge in Israel about running web operations drive them all to this "shortcut to hell".

Thanks for the comment.

Ori Lahav

Eran thanks for your second comment.

Interesting indeed.
Running on Virtual environment is a subject by itself that I agree I need more experience on to really talk smartly on this subject.

As I noted in the post itself. in order to scale well we are running several different solutions as databases and Data stores that are using different hardware specifications in order to work properly.
MySql have the specification I listed above but the Tokyo Tyrants are running with much less memory and over SSD drives.
So basically there is no "one size fits all" on the hardware level.
That is the main reason why I think virtual environment will be less efficient for us. I agree it is much cooler to run virtually but if I had to run the Tokyos on homogeneous VM like cloud gives me - I would have to use many many more machines.


I understand and know of others who have used specialized hardware such as SSDs but I personally don't like to hang the performance of my system on specialized hardware.

Specialized hardware machines and 32Gb machines cost significantly more than 16Gb machines with regular disks.

Almost all big players try to find their sweet spot machine configuration and use a lot of that same machine.

You can argue that it might require more machines to reach the same performance, but usually in most cases if you optimize it correctly and look at it from a different point of view you might be able to do the same without specialized hardware.

Like anything else in life, using the cloud is a trade off and the solution is probably a hybrid one with both cloud and dedicated servers.

The fact that all big players have some sort of virtualized environment (Amazon, RackspaceCloud, GoGrid and others virutalize hardware, while Google AppEngine virtualize the platform - not needing to worry about it) shows that there is something there.

Other than that, one of the key points you raised is the lack of experienced sysadmins. The fact is there is always a lack of such people and the key for them to be most effective is automation. That is where cloud makes things a lot more easier. Easier to provision more servers, easier to automate, etc.

Also, with the dynamic load on sites, having a virtualized environment means you can easily scale up and down your system, even automatically, since provisioning becomes a breeze.

Ori Lahav

I agree on some of what you said and disagree on other stuff.
This is getting into another discussion of virtualization, private clouds and public clouds that is different from the subject of the post.

I promise to post about it and call for a discussion.

custom essays

Linux is outdated already and Microsoft is all over the places.

The comments to this entry are closed.

Outbrain Zone

Sailing trips

    This is a Flickr badge showing public photos from Ori Lahav tagged with Gocek05. Make your own badge here.

Alona's Etsy Shop