Sea Breeze and Biz

Swimming against the (local) current

No, this is not one of my Sailing and Paddling posts. This one is about the Web industry in Israel.

I'm following up on posts by Michael Eisenberg and Yaniv Golan from last week. Their posts describes very much in details the reasons why it is not very likely that a $1B internet company will grow from the Israeli environment. It's a highly recommended reading, and the discussion on comments - even more.

Both Michael and Yaniv take that from the point of view of the software development (MS vs Open source discussion) but I want to give another point of view of the same phenomena.

Outbrain is an Open Source shop, we are running on Linux and our main programming language is Java. We are not using cloud services and building our own data-centers on co-locations in 3 different cities in the US. Totally bizarre in the Israeli scene.

Is it simple to do? probably not. Does it worth doing? Damn, yes!!! Just ask the big guys from over the ocean.

I guess you can read in Michael and Yaniv's posts why the aspects of Open Source is so crucial to your business scaling but I want to focus here on the operational aspects of a web company and why the "easy way" is sometimes tempting but deceiving.

Why Linux? 

The simple answer - I don't know how a Microsoft license look like, neither how does their invoice look like and I don't have to wast time negotiating with sleazy  MS salesman to cut the license price for my hundreds of servers. It is a big cost saving if you are thinking ahead towards the thousands of machines (which is usually where $1B company is).

Another reason - the Sys Admins are harder to find, but when you find them, its the type of people you really want on the boat with you. Those who can control the beast instead of fearing of it are the real masters. They are the people who will break any barrier on the way to the $1B company.

Why Open Source infrastructure ?

By Open Source infrastructure I mean systems like MySql, Cassandra, Tokyo Tyrant, memcached, MogileFS, etc... outbrain is a one big Zoo of those.

It appears that we are not the first company dealing with scaling a web product. Companies like google, Yahoo!, live-Journal, facebook, etc... did it before us and were generous enough to release their infrastructures to Open Source. They actually made the way easier for us. Sorry but I don't remember an internet Giant that is based on Microsoft/Oracle (except for maybe eBay but you don't want to take them as technological example).

Why Co-location and not Cloud?

Cost, Cost, Cost!!!

When you are small Cloud seem very charming and very cost efficient. you still don't know what your application will look like and what will be it's scaling factors so the scaling down is still an issue.
WATCH OUT!!! - its a honey trap.

In general, if you want to be a $1B company you have to start thinking like one from the first day. You have to start building the skills and talent of your team from the first day. you might not need to run your own datacenter but you should never get too much attached to the cloud service you are running on. Just because it will drain you out of $$$ when you grow.

I know most of you don't believe but let me give you an example:

outbrain have about 10 Mysql slaves. Each one with 16 cores 32GB RAM and 6x10k RPM drives in stripe. Amazon AWS doesn't have a machine in that scale that can support my needs. But lets take Amazon's "quadruple" instance and compare it to outbrain's machines in cost per year.

• Amazon's hourly rate is 7 times more expensive then our (stronger) machine.

• Amazon's reserved rate is 3 times more expensive then our (stronger) machine.

So where is the logic???

The Truth is that with your machines you can optimize better and better on both performance and cost which are the fundamentals of scaling. Your cost vs. growth curve becomes sub-linear while Cloud stays linear.

Why we don't see many Data center driven start ups in Israel? few reasons:

1. Its harder and there is a lack of that knowledge in Israel. But again, when you get to train the team to run on datacenter it is much more cost efficient.
2. there is a rumor that in the cloud you don't need Sys Admins. This is simply not true. Every system needs administration and you only fool yourself if you ignore the fact that instead of sys admin, your developers are doing all the setup and configurations of servers and infrastructure, instead of developing. In our experience only 1/4 of the time the ops team is dealing with hardware and networking, all the rest of the time they are dealing with the infrastructure which is software you need anyway.
3. VCs simply don't get it. Here is the paradox: When a VC is giving you $$$ they do it because they believe you will be a $1B company. On their next sentence they tell you - "move yourself to the cloud, it is the next hot thing, datacenters are part of the history and it scales beautifully". I hope the reasons I've listed above explains why you are going to throw their $$$ down the toilet once you will become what they wish you to become ($1B... remember?).

Again, building a web setup like Google, Facebook, Twitter, etc... is challenging. But you know what?

• It's more efficient.
• It builds better team that can handle greater challenges.
• It attracts the best talent that looks for challenges.

And... for Michael's point It builds a knowledge base in Israel.

If you read it and want to hear more - I'll be happy to assist.

Ori

with honor to Nachum Sharfam that the combination tech knowledge, entrepreneurship  and Zionism were always in his mind.

Ops and Scalability stuff

Hi again,

It's been long time since I have written anything on this blog. Lots have been going on in outbrain, mostly amazing stuff!!! Twitter have been filling lots of my sharing needs and put the blog aside for a while. So does our podcast which became successful and we keep for 18 months already.

However, from time to time the writing drives come up.

During the last year or so, I'm leading the operations team in outbrain, this role, especially in company that grows the business as fast as outbrain, has a great deal of scalability challenges which are fascinating to handle.  I will try to share with you some of it in the next few posts.

This time I want to share with you a site I recently bumped into (thanks Nathan).

http://scale.metaoptimize.com/ is a kind of a "stackoverflow" kinda site that is dealing only on Scalability/NoSQL/High Availability/all * computing/etc...

I want to share here an answer I gave there to this question:

Design for n+1 scaling upfront or not?

hey,

That's the big question and the short answer is "don't need to give a damn about scalable design".

The longer answer is: During product life cycle, especially in a new venture, you are zigzaging a lot. The product idea that you have today is probably very far from what it will end up to be. Why is that? because when you start a Start-up you are so damn dumb about what your users want. you just have a vague idea that make sense to some friends and investors. The real wisdom start flowing into the product only after few iterations where you were failing to get the product to "catch" user base. and as you know, product without user base doesn't need any scalability.

So having that there is no other way than making enough mistakes to get to one viable product, all you need to develop is the method to get a lean product out FAST. For that - scalability is the lowest priority task for you. As CTO of start-up that had to face (and still facing) scalability Challenges I can only tip you with one thing. Once you have an idea that this is going to be big, make sure you have the personnel that is capable in building and designing a scalable system.

Start-up is a roller-coaster - enjoy the ride!

Multiple Data Centers and the Scalability Step Function

This week outbrain launched the second Data Center in the US west coast.

Staying in one cozy Datacener is always more comfortable but there are lot's of reasons also to open another one as you grow.

I will name a few:

1.  Disaster recovery - you want to be able to still server your clients even if a 'plane crushed into your datacenter building' (not that such things happen :). 
2. Global load balancing and users proximity - you want to serve your clients from places that are close to them.
3. Data replication - you always have your data replicated to more then one physical location so it is harder to lose it.

I can probably name some more reasons but I want to write now about another interesting reason which is the scalability step function. usually when a web business scales, there are 2 ways to handle it. Scaling UP and Scaling OUT. By Scaling UP you always buy bigger, faster and more expensive machines that can handle your load and size of data. By Scaling OUT you build your application in such way that you can always add more machines to handle coming load.  The big problem of  Scaling UP it the "Step function" in cost. If your system top capacity is X and it cost Y, when you need to grow your system to 1.1X the next jump in cost is probably to 2 (or 4)Y. When scaling OUT, when the demand grow to 1.1X you only add 0.1Y to the cost and there is no need to put redundant cost.

What does this have to do with multiple Data Centers? Actually... A lot.

If you have the ability to manage more then one datacenter, you have the ability to scale your system OUT in much lower costs.

Let's see what happens in datacenters. you start with a single rack, then you expand to 2 then after the 3rd there is no rack in proximity  and you start wiring all around. Before the 5th rack you want all of your racks to be in proximity so you get a "cage" which is a leased area to put all your racks in proximity. (ooops - a step in the cost). then on the 30th rack you need a bigger space and you do a "migration project to space 4 times bigger (another big step).

Apart from the space you allocate for the servers, keeping them on the same LAN turns to be very costly in the bigger numbers. Connecting 100 servers to a LAN demands more then 1 switch and if you want ports redundancy it becomes even harder. then you jump to higher level switches that complicates the administration and when you get really big the next "Step" is core switches that makes Cisco's salesman smile and your CFO cry.

Instead of that, when you have the ability to manage multiple datacenters, when your setup hits it's limit - you just open another datacenter, can be even next door to your current one and you start racking this one with servers as you grow. Your step functions become smaller and most important, linear to your growth.

You can even shrink your business if you like. Lost some clients, did a business change or optimized your system? one datacenter became redundant? leave it. this is something you can't do when committing to one big space an after doing the network investment.

I guess you've got my point.

Lastly - I wan to thank 2 of our service providers that enabled this ability:

• Atlantic Metro Communication - our CoLocation provider in both sites.
• Cotendo - our global DNS provider.

They are both highly recommended.

Now everybody can be heard - OutLoud

This post is not for the big Co's. It's even not for the big media buyers.

It's for the small businesses, for the millions of one-man, two-man shows that runs their nice little business. These little businesses are doing great things but have very little chance to get nice exposure and expand the business.

I'm talking about Artists, Musicians, Writers, Technicians, Free Lance software engineers, eBay sellers, Etsy artists, etc... all these are paying one or two salaries but usually don't have the budget to buy the advertising money to get a banner in a news site.

What they usually have is good WOM (Word Of Mouth), can be a blog post written about them or about the work they do by some enthusiastic customer. When they see it they say: "WOW it's great to see somebody think that about my work, I wish every body could read it". Now its possible.

Now everybody can be heard - OutLoud!

Small businesses, Artists, Free Lancers, Internet sellers like eBay or Etsy,... everybody who gives products and services. If you have a good story written about you on the net. If the story is authentic and interesting to read. all you need to do is submit the link in outbrain site and for only $10 a month your story will be shown below relevant stories  on sites  like USAToday.com, Slate, Chicago Tribune and many many other sites and blogs.

The system will drive traffic to the story about you continuously and get more people to know you or your product. People that otherwise were never expose to the good service you give.

Now... you are heard! Say it OutLoud

Outbrain and tech support

One of the things we understood in a very early stage of the outbrain product life is the space we are acting in and the huge influence it can have on the company. We are in the zone of the reporters. We consider every blogger as a reporter with massive audience. The truth is that every blogger can start a bad WOM (word of mouth) on us which will be very hard to revert.
One of our biggest missions is "stay on the positive side of the WOM".
So we took few considerations to do it:

• We are monitoring our WOM closely. We have a google alert on us as well as twitter alert. So we know of any potential WOM, good or bad, at the time it starts.

• We put great effort on quality. Formal QA as well as software automated unit test that are running upon every commit to the SVN.

But as we all know, no software is 100% perfect and there will always be glitches. Especially if you are acting in such dynamic space as the blogosphere, there is no way to predict all the HTML variations to act on.
The way we handled that is to create a "support culture" within the company.
We initiated a support forum based on getsatisfection, which is a great tool for interacting with your support needed. We are all registered as mail subscribers for every issue being raised there. It includes CEO as well as the latest engineer that just joined. We are all equally responsible for our support responsiveness.
When needed, an engineer will drop anything he is currently doing in order to give support promptly.
We get very high appreciation from the bloggers on this.
The bottom line is that we convert potential criticizers to big fans of us. Since they are all reporters, we count on them to stand on our side for every nasty criticism that will come in the future.

Support is a tough, routine job which is basically not as cool and innovative as other design and development work. However, it is so rewarding when you get the warm words of a blogger that you were willing to assist him. We sometimes find ourselves helping bloggers with other nasty widgets that crap their blogs.

Our ability to give this great support, is mostly due to the team members themselves that took it as a personal mission. So thanks Daniel, Uriel, Tom, Gabi, Itai, John and mostly Rotem. You are all doing great job on this.

BTW: this post is written on my IPod touch. Also great story of customer support.

Blogs Discovery news

This morning an email was waiting for me from Oren, one of our researchers with a list of blogs that are semantically similar to my blog. On the top of the list were blogs I'm mostly familiar with and reading on a regular basis. But... I was delighted to find there at a pretty high place the blog called SeaWay with great articles and photos from the sailing and yachting zone.
Enjoy the new blog, as you can all enjoy posts and articles discovery from... outbrain.

Build the right team

We are now intensively hiring in outbrain and will extremely grow the team in the near future. As the head of the Israeli tech team, I took the lead on the hiring effort. Alen Stern wrote nicely about it here
and I agree that the higher manager possible should be the one to lead the hiring effort.  A co-founder and CTO is the most appropriate one to lead the effort of hiring tech people.

Good people are coming from many sources and I learned not to judge any of them.  I never say no to CVs coming from recruiters, nor head hunters. I judge the CVs by their content only.
We have a great team in outbrain. It is professionally balanced between experienced people and those who are 'young' and eager to learn. We have those who are family people and those who are young singles. We always encourage them all to have the right life/work balance.

I want to keep the larger team on that balance and looking for the right people to be contributers as well as contributed during their time in work.

The blogosphere also have some gems about picking the right people for your team.

Joel wrote a great post about it long ago. I call it the "Smart and get things done" criteria for recruiting. The people he is hiring are the people how have those 2 attributes together, they are smart and get things done.

Greg Linden, in one of his last posts quoted someone else about the people who are "Done, and get things smart". Those are pure gold to the team.
They do their work well but also improve stuff and get the rest of the team smarter.

I like it and advice everyone who is building a team or interviewing people to read them.

enjoy

In case you forgot... Outbrain is Hiring

In case you don't remember or hear about it the first time.

OUTBRAIN IS HIRING !!!

For our cool office in Natanya, we are looking for:

• Java / SQL / back-end engineers (both seniors and juniors).
• Java / javascript web developers.
• Operations / IT engineers.
• QA Engineers.

All should be people who are fun to work with as well as hard working people.
If you are a blogger or like the blogging zone... you are in the right place.

If you are only looking for the next place to get the pay-check... you are not our type.

So... to apply,  just contact us.
a comment here or visit to our Jobs page or all the ways to contact us are  relevant

"We want you as the new recruit!!!"

In the light of the "Castles and Ships" post i had before.
And the fact that, we, in outbrain are recruiting again.

I simply couldn't skip this YouTube video.

Thanks Hannit

If you are a type of a sailor, viking or simply a "fun to work with" talented programmer ... take a look at our Jobs page.

welcome aboard.