Swimming against the (local) current

No, this is not one of my Sailing and Paddling posts. This one is about the Web industry in Israel.

I'm following up on posts by Michael Eisenberg and Yaniv Golan from last week. Their posts describes very much in details the reasons why it is not very likely that a $1B internet company will grow from the Israeli environment. It's a highly recommended reading, and the discussion on comments - even more.

Both Michael and Yaniv take that from the point of view of the software development (MS vs Open source discussion) but I want to give another point of view of the same phenomena.

Outbrain is an Open Source shop, we are running on Linux and our main programming language is Java. We are not using cloud services and building our own data-centers on co-locations in 3 different cities in the US. Totally bizarre in the Israeli scene.

Is it simple to do? probably not. Does it worth doing? Damn, yes!!! Just ask the big guys from over the ocean.

I guess you can read in Michael and Yaniv's posts why the aspects of Open Source is so crucial to your business scaling but I want to focus here on the operational aspects of a web company and why the "easy way" is sometimes tempting but deceiving.

Why Linux? 

The simple answer - I don't know how a Microsoft license look like, neither how does their invoice look like and I don't have to wast time negotiating with sleazy  MS salesman to cut the license price for my hundreds of servers. It is a big cost saving if you are thinking ahead towards the thousands of machines (which is usually where $1B company is).

Another reason - the Sys Admins are harder to find, but when you find them, its the type of people you really want on the boat with you. Those who can control the beast instead of fearing of it are the real masters. They are the people who will break any barrier on the way to the $1B company.

Why Open Source infrastructure ?

By Open Source infrastructure I mean systems like MySql, Cassandra, Tokyo Tyrant, memcached, MogileFS, etc... outbrain is a one big Zoo of those.

It appears that we are not the first company dealing with scaling a web product. Companies like google, Yahoo!, live-Journal, facebook, etc... did it before us and were generous enough to release their infrastructures to Open Source. They actually made the way easier for us. Sorry but I don't remember an internet Giant that is based on Microsoft/Oracle (except for maybe eBay but you don't want to take them as technological example).

Why Co-location and not Cloud?

Cost, Cost, Cost!!!

When you are small Cloud seem very charming and very cost efficient. you still don't know what your application will look like and what will be it's scaling factors so the scaling down is still an issue.
WATCH OUT!!! - its a honey trap.

In general, if you want to be a $1B company you have to start thinking like one from the first day. You have to start building the skills and talent of your team from the first day. you might not need to run your own datacenter but you should never get too much attached to the cloud service you are running on. Just because it will drain you out of $$$ when you grow.

I know most of you don't believe but let me give you an example:

outbrain have about 10 Mysql slaves. Each one with 16 cores 32GB RAM and 6x10k RPM drives in stripe. Amazon AWS doesn't have a machine in that scale that can support my needs. But lets take Amazon's "quadruple" instance and compare it to outbrain's machines in cost per year.

• Amazon's hourly rate is 7 times more expensive then our (stronger) machine.

• Amazon's reserved rate is 3 times more expensive then our (stronger) machine.

So where is the logic???

The Truth is that with your machines you can optimize better and better on both performance and cost which are the fundamentals of scaling. Your cost vs. growth curve becomes sub-linear while Cloud stays linear.

Why we don't see many Data center driven start ups in Israel? few reasons:

1. Its harder and there is a lack of that knowledge in Israel. But again, when you get to train the team to run on datacenter it is much more cost efficient.
2. there is a rumor that in the cloud you don't need Sys Admins. This is simply not true. Every system needs administration and you only fool yourself if you ignore the fact that instead of sys admin, your developers are doing all the setup and configurations of servers and infrastructure, instead of developing. In our experience only 1/4 of the time the ops team is dealing with hardware and networking, all the rest of the time they are dealing with the infrastructure which is software you need anyway.
3. VCs simply don't get it. Here is the paradox: When a VC is giving you $$$ they do it because they believe you will be a $1B company. On their next sentence they tell you - "move yourself to the cloud, it is the next hot thing, datacenters are part of the history and it scales beautifully". I hope the reasons I've listed above explains why you are going to throw their $$$ down the toilet once you will become what they wish you to become ($1B... remember?).

Again, building a web setup like Google, Facebook, Twitter, etc... is challenging. But you know what?

• It's more efficient.
• It builds better team that can handle greater challenges.
• It attracts the best talent that looks for challenges.

And... for Michael's point It builds a knowledge base in Israel.

If you read it and want to hear more - I'll be happy to assist.

Ori

with honor to Nachum Sharfam that the combination tech knowledge, entrepreneurship  and Zionism were always in his mind.

Comments

[Nir Yeffet]

Well said. It is a honey trap, also because once you marry to the "cloud services" (for example Amazon's S3, rightscale's scripts, etc.) it is very hard to exit.

BTW, I love Tokyo Cabinet, Maybe I should write something better than Tokyo Tyrant? I can do real wonders with epoll :-)

[Eran]

I agree that you must have sysadmins, however I think ALL developers have some knowledge of administration. You have to know all of your stack bottom up and top to bottom to really understand how the system works, where are the bottlenecks and be able to really scale.

Regarding the cloud, the problem is not the cloud itself, its the platform. Amazon runs most of its own site on its own cloud platform and there are a lot of benefits for that.

Just consider a simple deployment of new version in the cloud vs. physical computers:

Cloud:
1) Start new instances with new code
2) Test them on real production config
3) Using the load balancer route more and more traffic to them. If there are problems you always have your old instances running.
4) If all is ok, move all traffic to new instances and shutdown the old ones.

Physical machines:
1) If you have spare machines or can downsize capacity for a while, install the new code there.
2) Test new code
3) Start moving traffic to new machines / upgraded machines
4) Upgrade the rest of the machines
5) If you have an issue rolling back is a bitch (at least more than shutting down a couple of instances in the cloud).

From a scaling point of view, your architecture shouldn't be dependent on the cloud at all. You should be able to migrate to physical machines (or just use a cloud provider such as GoGrid or Rackspace cloud that provide both cloud and physical hosting).

As you said, you need admins for the cloud, so the skill set stays and knowledge is still required. You just save, at least at first, the 1/4 time spent on networking and hardware.

If a VC tells you that must go to the cloud because its hot, you might want to reconsider taking this VCs money... :-)

[Eran]

Oh, and using Xen, KVM for the raw virtualization or the Eucalyptus project (which Ubuntu's private cloud is based upon) can give you the cloud environment you need.

Today I also heard that Rackspace are opensourcing their cloud tech called OpenStack to the wild. So things are going to be interesting.

[Ori Lahav]

Eran, I agree on all you have said.
Here is how we solved the deployment/live testing issue.

In outbrain we have few machines that are assigned to "staging" environment. they usually take live traffic and we are able to regulate traffic to them via our load balancer (BTW: HAProxy over Linux - also Open Source). new versions are deployed to the staging Environment and are measured for performance as well as our applications KPIs. once we are happy with it in staging we are releasing to the datacenters one by one so if there is a fault you can shift traffic to an old version datacenter while rolling back the new one.

one thing to have in mind regarding the "flexibility" of the cloud is that usually web applications, in order to get fast response, are keeping as much data as they can in RAM. this makes the cloud less attractive because loosing or giving up machines mean also giving up portion of the data that will hit your performance when you want to recover from it.
Basically if you need your machines to be live 24/356 and serve traffic, there is no reason to keep them in the cloud - it is simply not cost efficient.

As for the VCs, I agree but the problem is that entrepreneurs think the VCs knows better then them and the lack of knowledge in Israel about running web operations drive them all to this "shortcut to hell".

Thanks for the comment.
Ori

[Ori Lahav]

Eran thanks for your second comment.

Interesting indeed.
Running on Virtual environment is a subject by itself that I agree I need more experience on to really talk smartly on this subject.

As I noted in the post itself. in order to scale well we are running several different solutions as databases and Data stores that are using different hardware specifications in order to work properly.
Examples:
MySql have the specification I listed above but the Tokyo Tyrants are running with much less memory and over SSD drives.
So basically there is no "one size fits all" on the hardware level.
That is the main reason why I think virtual environment will be less efficient for us. I agree it is much cooler to run virtually but if I had to run the Tokyos on homogeneous VM like cloud gives me - I would have to use many many more machines.

[Eran]

I understand and know of others who have used specialized hardware such as SSDs but I personally don't like to hang the performance of my system on specialized hardware.

Specialized hardware machines and 32Gb machines cost significantly more than 16Gb machines with regular disks.

Almost all big players try to find their sweet spot machine configuration and use a lot of that same machine.

You can argue that it might require more machines to reach the same performance, but usually in most cases if you optimize it correctly and look at it from a different point of view you might be able to do the same without specialized hardware.

Like anything else in life, using the cloud is a trade off and the solution is probably a hybrid one with both cloud and dedicated servers.

The fact that all big players have some sort of virtualized environment (Amazon, RackspaceCloud, GoGrid and others virutalize hardware, while Google AppEngine virtualize the platform - not needing to worry about it) shows that there is something there.

Other than that, one of the key points you raised is the lack of experienced sysadmins. The fact is there is always a lack of such people and the key for them to be most effective is automation. That is where cloud makes things a lot more easier. Easier to provision more servers, easier to automate, etc.

Also, with the dynamic load on sites, having a virtualized environment means you can easily scale up and down your system, even automatically, since provisioning becomes a breeze.

[Ori Lahav]

Eran
I agree on some of what you said and disagree on other stuff.
This is getting into another discussion of virtualization, private clouds and public clouds that is different from the subject of the post.

I promise to post about it and call for a discussion.

[custom essays]

Linux is outdated already and Microsoft is all over the places.