LinkedIn is a Wall Street darling, its stock up more than threefold in two years on soaring revenue, spiking profits, and seven straight quarters beating bankers’ estimates. But LinkedIn's success isn't just about numbers: an impressive acceleration of LinkedIn's product cycle and a corresponding revolution in how LinkedIn writes software is a huge component in the company's winning streak.
Much of LinkedIn’s success can be traced to changes made by Kevin Scott, the senior vice president of engineering and longtime Google veteran lured to LinkedIn in Feb. 2011, just before the buttoned-down social network went public. It was Scott and his team of programmers who completely overhauled how LinkedIn develops and ships new updates to its website and apps, taking a system that required a full month to release new features and turning it into one that pushes out updates multiple times per day.
The result can be seen today as LinkedIn releases a smart set of new features for recruiters, including an intelligent "people you should hire" suggestion box. And it can be seen in all the other products LinkedIn has banged out over the past year, including revamped company pages, overhauled notifications, a redesigned homepage, comments and likes on news pages, iPad and Windows Phone apps, revamped profile pages, a job listings app, and blogging features. Just this week, Linkedin rolled out a feature cribbed from Facebook that lets users hotlink their friends in status updates.
“Without having done all this work to change how we built our software,” says Scott, “it literally would be impossible for us to be building endorsements, and the new influencers product, and the new version of profile and the things that are happening in mobile, and the upgrades we made to the recruiter product, and all of these dozens of significant changes that are rolling out.”
LinkedIn's newly-adopted software development methodology is known as "continuous deployment." Under continuous deployment, a developer writes new code in tidy, discrete little chunks and quickly checks each chunk into the main line of software shared amongst all LinkedIn developers, a line known as “trunk” within the software version control systems standard in the tech industry. Newly-added code is subjected to an elaborate series of automated tests designed to weed out any bugs. Once the code passes the tests it is merged into trunk and cataloged in a system that shows managers what features are ready to go live on the site or in new versions of LinkedIn’s apps.
LinkedIn’s prior system of software development was more traditional, involving software “branches” forked off from trunk and developed in parallel over a period of weeks or days. A developer would finish a big batch of code corresponding to some feature and then lobby for this feature branch to be merged into trunk. Once merged into trunk, the feature would again need to be tested to ensure it did not break any of the other new code checked into trunk at the same time. Bugs and outright broken software are common under this so-called "feature branch" system, since typically several big batches of code, each written in isolation by a separate team, are merged into trunk at once. To avoid such meltdowns, managers tended to tightly limit the number and scope of new features mashed together each month, slowing down a company's development cycle.
“It was a pretty big risk the business took, to say to its engineering team, you’re going to run across a bridge and burn it behind you.”Shifting from feature-branch-based development to the new continuous deployment system required halting all development for two months as LinkedIn trained staff, migrated old code, and built out the automated tools it needed to make the new system work.
“It was a pretty big risk the business took,” says Scott, “to look at its engineering team and say, ‘we’re going to completely change the way we do software… and somewhere in the middle of this two-month process you’re going to run across a bridge and burn it behind you.”
LinkedIn is hardly the only company to use continuous deployment. Scott had experience with the system from prior gigs, and other internet companies have embraced the practice as well, including handcrafted-goods marketplace Etsy and Facebook. But LinkedIn’s big switch to continuous deployment has been linked to very concrete and visible financial success, helping lend credence to the practice and potentially helping to accelerate the delivery of software across the tech industry.
For Scott, the move to continuous deployment was about solving concrete problems rather than spreading a doctrine.
"We had to go from this model where developers were developing their code in relative isolation and then raising their hand and say, 'please integrate my feature branch into the release branch and test it and qualify it for me and push it out to the site in the appropriate release window,'" Scott says. "We wanted to be at the point where... as soon as they were checking in their code... it was qualified and releasable... that anything sitting in trunk must be releasable at any point in time, and if it's not releasable it's just as significant as a site emergency. Stop all forward software development and everyone is all hands on deck to get trunk fixed."
The experiment seems to be going well. Scott made his comments in our last interview, when he also outlined LinkedIn's system for nurturing employee side projects. Since then, the LinkedIn software barrage has continued smoothly -- and the stock has shot up 61 percent.
