Sea Breeze and Biz

Weekend of Kayaking in Connecticut

It all began with the lousy political situation we have with Turkey that is kinda preventing me from spending the regular sailing vacation. I tried to find appropriate substitute in the form of paddling. The paddling club trips abroad sounds like a super fun thing but they require longer engagement so I ended up combining a business trip to NYC with paddling around. Hadas, our club instructor, got many kayaking friends in the US so she did some mail introductions. The one that picked up the glove was Gerry Polansky, a paddling instructor from Connecticut. We had a long mail exchange to close all the trip details and we come up with something that suited my needs and Gerry’s schedule.

My trip to started with 2 working days in NY and on Friday afternoon I found myself on the train to New London Connecticut. Trains in this area are far more specious and convenient then airplanes and the first part of the ride during daylight amazed me by the green area and blue water coves and rivers we crossed. Good sign for what was coming next.

 

Saturday.

The next morning Gerry picked me up from the hotel. We had short introduction time and we hit the road towards nice little town a bit north of New London where we were going to launch the boats. Small town, houses on the waterfront and tons of boats on the water. All the rest is the Green, Green, Green surrounding everything. Coming from end of summer in Israel makes it totally “refreshing”.

Then the other guys started coming in. Experienced paddlers with their boats that looked very high techish and nice to ride. They were Steve, Ron, Carolyn and Nick which is the designer and builder of Guilmount Kayaks. Steve and few other people from our club in Israel bought plans from him and build beautiful well designed boats from it.

It took a while to get organized but we launched and started paddling toward Fisher’s Island, an Island ~4 miles out the Connecticut shore, place full of rich people houses and land that belongs to the navy so much of it is very much preserved. The paddling along the island was nice ride on the way to real fun spot.

At the south-western tip of the Island there is mouth of the Long Island sound. The place where the tide water are coming in and flushing out. That was the place where I learned the term ‘Race’ for a place where the Tidal Streams are running. It’s hard to see it in the pictures but this place is a fun washing machine. It’s a 200 meter wide area of water rushing from the one side creating Tidal waves to the other side. My instincts are that when you are surfing on a wave it carries you forward, but here the situation is completely different/ you are surfing the waves but if you are not constantly paddling the race carries you backward. For people used to coastal surf this is nothing like you knew before and totally different experience which I was very happy to have. The guys From the group were kind enough to watch my back as all this was pretty new to me but I managed to hold very nicely in these conditions and stay up all the time. Well except for one roll that I did to feel more comfortable in the water there. Right in the middle of the race there was a very nice lighthouse built on a rock . that supplied a nice calm spots to take some rest from the race, have a snack and even climb to the lighthouse platform and take some pictures. After some more playing in the race we headed back to Fisher Island shore for a lunch. We just found a spot with 2 logs, we had out lunch and took about 1 ½ hour of lazing in the sun chatting and throwing stones at other stones on the other log (next to Gerry:)

When it felt like we just packed everything and headed back to the Connecticut coast along with tons of traffic of power and sailing boats. Eventually we got back to our original launching spot, spent some fun time in the water and then packed the boats. We did not leave till we had nice afternoon meal in the club there with a lot of sea food water beer and tons of bread. Right what we needed after such nice paddling.

I think it will be hard to explain in words the fun I had with this group of fun and interesting people, and the generous hospitality and care I’ve got during this day. Apparently there is something unique and in common with all of us Kayaking people all around the world.

Right after Gerry brought me back to the hotel, I went to the New London town to see it in day light. It is a fun town to be in a summer weekend.

Want to locate this on a map? here you are.

 

Sunday.

The second day, Gerry picked me up with his car and a trailer with 5 other boats on it. This was the class day. We went into the beautiful Powers Lake where the water are clear and the “relatively” warm for Gerry to give his Level 2 class. We had there Dave, Susan and Kate which are taking this class for the second time and Steve and Pat that took a paddling class for the first time.

It was also the first time for me to take a formal skills class but eventually from that many years of paddling I already knew the basics, or as Gerry says, “he’s got the theory”.

The Class covered a lot of stuff that was new to me even if they are basic. I learned methods for rescues and self rescues. I learned nice hints about towing. This was the first time I really learned bracing (although after all these years this comes naturally). It was the first time for me to learn side stroke (paddling sideways) and bow tearing which was pretty cool.

We had nice lunch break where I gave the guys the “special treat” of Arabic coffee cooked on Gerry’s famous Primus Stove. I must say the guys were initially terrified by the way it looks and how Muddy it is comparing to the (lousy) American coffee but after a sip or two the seem to actually find it tasty.

This was an excellent day with a great chance for me to learn A LOT from someone like Gerry and also to give the people some of my knowledge and experience for all of us to get better.

We finished the day with delicious fruits on the way and nice Thai meal me and Gerry had before heading to the train station.

Wanna see where the lake is: it's here:

 

To Summarize: This was not a good compensation for the Turkey vacation I missed but a GREAT weekend with GREAT people, paying ‘love’ to the water again such GREAT scenery. Tons of thanks To Gerry.

Lots of pictures here:

Paddling in CT

ILTechTalks - Clarification

I want to thank you all for the support for the Israeli Tech Talks Initiative.

While talking to some of you, I figured out that many of you did not get the suggested form of the talks.

I know many are used to gatherings/meetings/conferences/... where you all seat and listen to a talk but this is NOT the suggested form.

Gatherings like I mentioned above are good for "teasing" us to "think" about new technology or methodology. Usually attendees to such gatherings are VP R&D, CTOs, Lead Developers but not all your development/tech team. Those that should implement and work with technology are not being engaged because they don't attend the talk. It also usually keep the talk on the general (management) level and real life questions, hurdles and challenges  are left out.

What I was suggesting is what I find as a better way to drive knowledge to the actual people that will do the job. Pick a talk and a speaker and bring him in-house to your company to talk to your team.  I found it as much better way to engage team members and "fire them up" towards a new technology or methodology. That way you also give them the ability to ask the questions that involve the day-to-day aspects and be sure they know get sufficient answers.

What we will also create is a network of knowledge that can result in calls like - "Hey Ran, I have few questions about my Cassandra setup, any chance to have a coffee and talk it over?".

I know, most information can be found out there in forums, mailing lists, StackOverflow, ServerFault, just google your question and you got it. But the ability to have a cup of coffee with an expert that went through your hurdles, in Hebrew and work things out is sometimes life saving.

I think we have the knowledge here in Israel, but we just don't know where to find it.

ILTechTalks is that kind of place where you can find other companies that have been through some technical challenges, solved them and are willing to share their knowledge with you and your team.

All you need to do is:

Go To our site.

If you have (and I know every company have at least one thing they see as their "craftsmanship") and list a talk.

If you want to ask for a talk you can do it here.

the wiki is invitation based so please register to our mailing list and ask for invitation.

Thanks and good luck to all of us.

Ori

Israeli Internet Tech Knowledge Exchange

I'm following a post I wrote few weeks ago in response to Michael Eisenberg's blog post.

One of the things I was thinking about is Knowledge and Experience.

I have the feeling Internet Start-ups in Israel do have the Human ability to generate a successful product and methodologies that are now embraced by Silicon Vally companies like Facebook and Flickr are much more suitable to the Israeli culture then to the American culture.   What I think we are missing is the spirit of a Tech Hub.

We are all looking at what the successful companies are doing in the US but we don't really have the ability to seat with a lead engineer in Facebook for a beer and talk about how they do stuff, or with VP Ops of twitter about their deployment and monitoring tools. etc...

The fact is, there is some knowledge here in Israel and there are some successful internet companies that run medium-big internet set-ups. There is some things we can learn from each other.

At outbrain, every Thursday afternoon we are doing a Tech Talk. We Gather in the Wii room, Take a coffee and listen to a technological talk brought by one of the team members or somebody from out side the company.

Recently we did something a bit different. During my visit in California in June, I gave a talk about scalability at Kaching and in exchange - during July Eishay Smith gave us a great talk in Outbrain office in Natanya about Continues Deployment.

I don't see any reason why we can't learn from each other. Some times you avoid doing the right things just because it seems complicated and frightening. Shifting your company to another technology which is out of your comfort zone sometimes sounds as a risk and risk averse people are avoiding it and by that miss big opportunities. usually knowing that somebody have been through this and succeeded, as well as being able to talk to him, ask questions and maybe share codes and examples can make the whole difference.

All we need is the ability to share knowledge and self educate our industry to share and help each other.

I'm picking the glove here and suggesting few subjects that we can share in outbrain:

• Outbrain's view of Scalability. (I will probably take that)
• Managing Data Center as oppose to cloud based set-up.
• Building internet company on top of Open Source only.
• NoSql and Cassandra (Ran is the master of this)
• Maven as Build tool (Ran)
• Javascript widgets and some of our learning.

We probably have some more.

We would be very happy to get some talk in exchange but it is not a condition.

If there will be a big need for exchanging talks we will probably make it more organized and set a site where you can all suggest talks and invite people to talk with your teams.

What do you say? Want to exchange?

UPDATE: I created a site for hosting all the talks - it's a wiki so please add your talk in there.

UPDATE: Yael Vaya has agreed to help with this venture - THANKS!!!

We also opened a Google Group for this here - let us know if you want an invite to update the site.

Greening our data centers

As I mentioned before outbrain is growing and we are extending our data centers set-up significantly.

This means high investment in hardware but also an increase to our recurring costs.  Recurring costs are those you will pay month by month in order to keep the those servers running. These are usually the space and electricity that you are consuming.

I want to talk today about the electricity (power) costs and how the new technologies we are choosing are combining the costs cutting motivation with making our planet a better and healthier place.

Usually when running a datacenter you are buying space and power from your collocation provider. The costs are per the power stripes you get around 20-30AMP power circuits and you pay for one as if you fully utilized it.  So... even if your machines are powered off - you still pay the same amount of $$$ monthly.

It happened to us that 2 of our main suppliers have advanced technologically to allow us to get better.

DELL - came out with their C series that let you combine 4 nodes into one 2U physical chassis.

AMC - our co-location vendor shifted to a "pay per use" power billing system on their new data-centers.

Let's start with the Hardware:

All Intel Nahelem architecture is much more power efficient and makes power consumption much more linear to the load on the machines. On top of that, due to more efficient  packaging the C series are even more power effective.

Here is the evolution of DELL's machines as reflected on outbrain's standard application machine:

• PE1950 - uses 2.8 amp
• R610      - uses 1.38 amp
• R410      - uses 1.31 amp
• 1 node within C6100 - uses 0.98 amp !!!

It means over ~2 years we have managed to reduce the power consumption of the machines we use to nearly a third just by using DELL's cutting edge technology. Again, when you look at the bigger picture, buying the newest stuff is proven more cost effective and... GREEN.

What effort you will see by getting more power effective machines if the power costs are always the same?

Here comes your co-location vendor billing method to help.  AMC, our co-location vendor was thinking ahead and in their new data centers in NYC and Chicago have already implemented that. 

Basically you pay by the amp, so you can have as many machines as you want set-up racked and cabled but you can turn them off to reduce power consumption. Even better, you are encouraged to use more effective hardware in order to reduce your recurring costs and by that... help GREENING the planet.

It's a cycle where everybody benefit from and as a by product we help reducing the global warming. I like it when vendors see that.

Well done DELL and AMC.

ohhh.... and, we are hiring Sys Admin in NYC - so, if you want to be part of this - take a look here.

Swimming against the (local) current

No, this is not one of my Sailing and Paddling posts. This one is about the Web industry in Israel.

I'm following up on posts by Michael Eisenberg and Yaniv Golan from last week. Their posts describes very much in details the reasons why it is not very likely that a $1B internet company will grow from the Israeli environment. It's a highly recommended reading, and the discussion on comments - even more.

Both Michael and Yaniv take that from the point of view of the software development (MS vs Open source discussion) but I want to give another point of view of the same phenomena.

Outbrain is an Open Source shop, we are running on Linux and our main programming language is Java. We are not using cloud services and building our own data-centers on co-locations in 3 different cities in the US. Totally bizarre in the Israeli scene.

Is it simple to do? probably not. Does it worth doing? Damn, yes!!! Just ask the big guys from over the ocean.

I guess you can read in Michael and Yaniv's posts why the aspects of Open Source is so crucial to your business scaling but I want to focus here on the operational aspects of a web company and why the "easy way" is sometimes tempting but deceiving.

Why Linux? 

The simple answer - I don't know how a Microsoft license look like, neither how does their invoice look like and I don't have to wast time negotiating with sleazy  MS salesman to cut the license price for my hundreds of servers. It is a big cost saving if you are thinking ahead towards the thousands of machines (which is usually where $1B company is).

Another reason - the Sys Admins are harder to find, but when you find them, its the type of people you really want on the boat with you. Those who can control the beast instead of fearing of it are the real masters. They are the people who will break any barrier on the way to the $1B company.

Why Open Source infrastructure ?

By Open Source infrastructure I mean systems like MySql, Cassandra, Tokyo Tyrant, memcached, MogileFS, etc... outbrain is a one big Zoo of those.

It appears that we are not the first company dealing with scaling a web product. Companies like google, Yahoo!, live-Journal, facebook, etc... did it before us and were generous enough to release their infrastructures to Open Source. They actually made the way easier for us. Sorry but I don't remember an internet Giant that is based on Microsoft/Oracle (except for maybe eBay but you don't want to take them as technological example).

Why Co-location and not Cloud?

Cost, Cost, Cost!!!

When you are small Cloud seem very charming and very cost efficient. you still don't know what your application will look like and what will be it's scaling factors so the scaling down is still an issue.
WATCH OUT!!! - its a honey trap.

In general, if you want to be a $1B company you have to start thinking like one from the first day. You have to start building the skills and talent of your team from the first day. you might not need to run your own datacenter but you should never get too much attached to the cloud service you are running on. Just because it will drain you out of $$$ when you grow.

I know most of you don't believe but let me give you an example:

outbrain have about 10 Mysql slaves. Each one with 16 cores 32GB RAM and 6x10k RPM drives in stripe. Amazon AWS doesn't have a machine in that scale that can support my needs. But lets take Amazon's "quadruple" instance and compare it to outbrain's machines in cost per year.

• Amazon's hourly rate is 7 times more expensive then our (stronger) machine.

• Amazon's reserved rate is 3 times more expensive then our (stronger) machine.

So where is the logic???

The Truth is that with your machines you can optimize better and better on both performance and cost which are the fundamentals of scaling. Your cost vs. growth curve becomes sub-linear while Cloud stays linear.

Why we don't see many Data center driven start ups in Israel? few reasons:

1. Its harder and there is a lack of that knowledge in Israel. But again, when you get to train the team to run on datacenter it is much more cost efficient.
2. there is a rumor that in the cloud you don't need Sys Admins. This is simply not true. Every system needs administration and you only fool yourself if you ignore the fact that instead of sys admin, your developers are doing all the setup and configurations of servers and infrastructure, instead of developing. In our experience only 1/4 of the time the ops team is dealing with hardware and networking, all the rest of the time they are dealing with the infrastructure which is software you need anyway.
3. VCs simply don't get it. Here is the paradox: When a VC is giving you $$$ they do it because they believe you will be a $1B company. On their next sentence they tell you - "move yourself to the cloud, it is the next hot thing, datacenters are part of the history and it scales beautifully". I hope the reasons I've listed above explains why you are going to throw their $$$ down the toilet once you will become what they wish you to become ($1B... remember?).

Again, building a web setup like Google, Facebook, Twitter, etc... is challenging. But you know what?

• It's more efficient.
• It builds better team that can handle greater challenges.
• It attracts the best talent that looks for challenges.

And... for Michael's point It builds a knowledge base in Israel.

If you read it and want to hear more - I'll be happy to assist.

Ori

with honor to Nachum Sharfam that the combination tech knowledge, entrepreneurship  and Zionism were always in his mind.

Ops and Scalability stuff

Hi again,

It's been long time since I have written anything on this blog. Lots have been going on in outbrain, mostly amazing stuff!!! Twitter have been filling lots of my sharing needs and put the blog aside for a while. So does our podcast which became successful and we keep for 18 months already.

However, from time to time the writing drives come up.

During the last year or so, I'm leading the operations team in outbrain, this role, especially in company that grows the business as fast as outbrain, has a great deal of scalability challenges which are fascinating to handle.  I will try to share with you some of it in the next few posts.

This time I want to share with you a site I recently bumped into (thanks Nathan).

http://scale.metaoptimize.com/ is a kind of a "stackoverflow" kinda site that is dealing only on Scalability/NoSQL/High Availability/all * computing/etc...

I want to share here an answer I gave there to this question:

Design for n+1 scaling upfront or not?

hey,

That's the big question and the short answer is "don't need to give a damn about scalable design".

The longer answer is: During product life cycle, especially in a new venture, you are zigzaging a lot. The product idea that you have today is probably very far from what it will end up to be. Why is that? because when you start a Start-up you are so damn dumb about what your users want. you just have a vague idea that make sense to some friends and investors. The real wisdom start flowing into the product only after few iterations where you were failing to get the product to "catch" user base. and as you know, product without user base doesn't need any scalability.

So having that there is no other way than making enough mistakes to get to one viable product, all you need to develop is the method to get a lean product out FAST. For that - scalability is the lowest priority task for you. As CTO of start-up that had to face (and still facing) scalability Challenges I can only tip you with one thing. Once you have an idea that this is going to be big, make sure you have the personnel that is capable in building and designing a scalable system.

Start-up is a roller-coaster - enjoy the ride!

#memthem - Israeli Memorial Day

This week is the Israeli Memorial Day for all those who sacrifice their life for the defense of the state of Israel and those who were killed in terrorist attacks and wars.
Unfortunately we have too many of those in Israel and this day is touching almost anyone of us here.
I was thinking, if any one of us will contribute a little twitt with the name of the person to remember and a link to a site, blog post, web page or even upload a picture to twitpic. Just add the tag #memthem.

I believe that way we can create a very big memory album for those we honor their memory. 

to view the memory album simply search for the #memthem on twitter

Multiple Data Centers and the Scalability Step Function

This week outbrain launched the second Data Center in the US west coast.

Staying in one cozy Datacener is always more comfortable but there are lot's of reasons also to open another one as you grow.

I will name a few:

1.  Disaster recovery - you want to be able to still server your clients even if a 'plane crushed into your datacenter building' (not that such things happen :). 
2. Global load balancing and users proximity - you want to serve your clients from places that are close to them.
3. Data replication - you always have your data replicated to more then one physical location so it is harder to lose it.

I can probably name some more reasons but I want to write now about another interesting reason which is the scalability step function. usually when a web business scales, there are 2 ways to handle it. Scaling UP and Scaling OUT. By Scaling UP you always buy bigger, faster and more expensive machines that can handle your load and size of data. By Scaling OUT you build your application in such way that you can always add more machines to handle coming load.  The big problem of  Scaling UP it the "Step function" in cost. If your system top capacity is X and it cost Y, when you need to grow your system to 1.1X the next jump in cost is probably to 2 (or 4)Y. When scaling OUT, when the demand grow to 1.1X you only add 0.1Y to the cost and there is no need to put redundant cost.

What does this have to do with multiple Data Centers? Actually... A lot.

If you have the ability to manage more then one datacenter, you have the ability to scale your system OUT in much lower costs.

Let's see what happens in datacenters. you start with a single rack, then you expand to 2 then after the 3rd there is no rack in proximity  and you start wiring all around. Before the 5th rack you want all of your racks to be in proximity so you get a "cage" which is a leased area to put all your racks in proximity. (ooops - a step in the cost). then on the 30th rack you need a bigger space and you do a "migration project to space 4 times bigger (another big step).

Apart from the space you allocate for the servers, keeping them on the same LAN turns to be very costly in the bigger numbers. Connecting 100 servers to a LAN demands more then 1 switch and if you want ports redundancy it becomes even harder. then you jump to higher level switches that complicates the administration and when you get really big the next "Step" is core switches that makes Cisco's salesman smile and your CFO cry.

Instead of that, when you have the ability to manage multiple datacenters, when your setup hits it's limit - you just open another datacenter, can be even next door to your current one and you start racking this one with servers as you grow. Your step functions become smaller and most important, linear to your growth.

You can even shrink your business if you like. Lost some clients, did a business change or optimized your system? one datacenter became redundant? leave it. this is something you can't do when committing to one big space an after doing the network investment.

I guess you've got my point.

Lastly - I wan to thank 2 of our service providers that enabled this ability:

• Atlantic Metro Communication - our CoLocation provider in both sites.
• Cotendo - our global DNS provider.

They are both highly recommended.

Now everybody can be heard - OutLoud

This post is not for the big Co's. It's even not for the big media buyers.

It's for the small businesses, for the millions of one-man, two-man shows that runs their nice little business. These little businesses are doing great things but have very little chance to get nice exposure and expand the business.

I'm talking about Artists, Musicians, Writers, Technicians, Free Lance software engineers, eBay sellers, Etsy artists, etc... all these are paying one or two salaries but usually don't have the budget to buy the advertising money to get a banner in a news site.

What they usually have is good WOM (Word Of Mouth), can be a blog post written about them or about the work they do by some enthusiastic customer. When they see it they say: "WOW it's great to see somebody think that about my work, I wish every body could read it". Now its possible.

Now everybody can be heard - OutLoud!

Small businesses, Artists, Free Lancers, Internet sellers like eBay or Etsy,... everybody who gives products and services. If you have a good story written about you on the net. If the story is authentic and interesting to read. all you need to do is submit the link in outbrain site and for only $10 a month your story will be shown below relevant stories  on sites  like USAToday.com, Slate, Chicago Tribune and many many other sites and blogs.

The system will drive traffic to the story about you continuously and get more people to know you or your product. People that otherwise were never expose to the good service you give.

Now... you are heard! Say it OutLoud

Wor(l)d of Mouth

Found the term "World of Mouth" in a presentation in this post by Gal Mor (Hebrew).

The slides in the presentation moves very quickly but you can clearly see the section speaking about WOM marketing. That's basically what we do in outbrain.

Almost every product now has WOM speaking about it, almost every business have it's group of fans that share their excitement on the social media.

All you have to do to promote the content speaking about your great product or service is to send it to us in outbrain and for only $10/month we will promote it for relevant audience.

here is the presentation, It tells better then me why it's the right thing to do.