No, this is not one of my Sailing and Paddling posts. This one is about the Web industry in Israel.
I'm following up on posts by Michael Eisenberg and Yaniv Golan from last week. Their posts describes very much in details the reasons why it is not very likely that a $1B internet company will grow from the Israeli environment. It's a highly recommended reading, and the discussion on comments - even more.
Both Michael and Yaniv take that from the point of view of the software development (MS vs Open source discussion) but I want to give another point of view of the same phenomena.
Outbrain is an Open Source shop, we are running on Linux and our main programming language is Java. We are not using cloud services and building our own data-centers on co-locations in 3 different cities in the US. Totally bizarre in the Israeli scene.
Is it simple to do? probably not. Does it worth doing? Damn, yes!!! Just ask the big guys from over the ocean.
I guess you can read in Michael and Yaniv's posts why the aspects of Open Source is so crucial to your business scaling but I want to focus here on the operational aspects of a web company and why the "easy way" is sometimes tempting but deceiving.
The simple answer - I don't know how a Microsoft license look like, neither how does their invoice look like and I don't have to wast time negotiating with sleazy MS salesman to cut the license price for my hundreds of servers. It is a big cost saving if you are thinking ahead towards the thousands of machines (which is usually where $1B company is).
Another reason - the Sys Admins are harder to find, but when you find them, its the type of people you really want on the boat with you. Those who can control the beast instead of fearing of it are the real masters. They are the people who will break any barrier on the way to the $1B company.
Why Open Source infrastructure ?
By Open Source infrastructure I mean systems like MySql, Cassandra, Tokyo Tyrant, memcached, MogileFS, etc... outbrain is a one big Zoo of those.
It appears that we are not the first company dealing with scaling a web product. Companies like google, Yahoo!, live-Journal, facebook, etc... did it before us and were generous enough to release their infrastructures to Open Source. They actually made the way easier for us. Sorry but I don't remember an internet Giant that is based on Microsoft/Oracle (except for maybe eBay but you don't want to take them as technological example).
Why Co-location and not Cloud?
Cost, Cost, Cost!!!
When you are small Cloud seem very charming and very cost efficient. you still don't know what your application will look like and what will be it's scaling factors so the scaling down is still an issue.
WATCH OUT!!! - its a honey trap.
In general, if you want to be a $1B company you have to start thinking like one from the first day. You have to start building the skills and talent of your team from the first day. you might not need to run your own datacenter but you should never get too much attached to the cloud service you are running on. Just because it will drain you out of $$$ when you grow.
I know most of you don't believe but let me give you an example:
outbrain have about 10 Mysql slaves. Each one with 16 cores 32GB RAM and 6x10k RPM drives in stripe. Amazon AWS doesn't have a machine in that scale that can support my needs. But lets take Amazon's "quadruple" instance and compare it to outbrain's machines in cost per year.
- Amazon's hourly rate is 7 times more expensive then our (stronger) machine.
- Amazon's reserved rate is 3 times more expensive then our (stronger) machine.
So where is the logic???
The Truth is that with your machines you can optimize better and better on both performance and cost which are the fundamentals of scaling. Your cost vs. growth curve becomes sub-linear while Cloud stays linear.
Why we don't see many Data center driven start ups in Israel? few reasons:
- Its harder and there is a lack of that knowledge in Israel. But again, when you get to train the team to run on datacenter it is much more cost efficient.
- there is a rumor that in the cloud you don't need Sys Admins. This is simply not true. Every system needs administration and you only fool yourself if you ignore the fact that instead of sys admin, your developers are doing all the setup and configurations of servers and infrastructure, instead of developing. In our experience only 1/4 of the time the ops team is dealing with hardware and networking, all the rest of the time they are dealing with the infrastructure which is software you need anyway.
- VCs simply don't get it. Here is the paradox: When a VC is giving you $$$ they do it because they believe you will be a $1B company. On their next sentence they tell you - "move yourself to the cloud, it is the next hot thing, datacenters are part of the history and it scales beautifully". I hope the reasons I've listed above explains why you are going to throw their $$$ down the toilet once you will become what they wish you to become ($1B... remember?).
Again, building a web setup like Google, Facebook, Twitter, etc... is challenging. But you know what?
- It's more efficient.
- It builds better team that can handle greater challenges.
- It attracts the best talent that looks for challenges.
And... for Michael's point It builds a knowledge base in Israel.
If you read it and want to hear more - I'll be happy to assist.
with honor to Nachum Sharfam that the combination tech knowledge, entrepreneurship and Zionism were always in his mind.