Update: See the new resource site for Trunk-Based Development called, err, TrunkBasedDevelopment.com and make sure to tell your colleagues about it and this high-throughput branching model.

A week ago, I wrote an article on Trunk-Based Development (TBD) at Facebook, but there’s newer source material. At InfoQ’s QCon this time, Chuck Rossi presented newer TBD practices and release workflow at Facebook. To be fair, there’s a lot of similarities to the May ‘11 presentation that he gave. At this stage I’m sure he could give the entire presentation from memory, with only a whiteboard and a couple of dry-erase pens.

There is a subtlety that I missed in the first article, which I’ll correct in this one. I did not outline their use of ‘latest’ (integration) branch, albeit transitory. Many corporate dev teams doing a trunk model would implement the trunk + numbered branch-for-release model I outlined before. Facebook is slightly different, but then they are not enterprise.

That is also some ‘new’ that Facebook are doing since that presentation. See below

The improved branching diagram

I’d previously suggested that they numbered release branches. That’s a common trunk based development practice, but it appears they have some fixed branch names, that keep cycling in some form of weekly Groundhog Day. As well as being easy for humans to remember, it also makes it easier for scripts to know the from and to branch for a merge. e.g. from “trunk”, to “production”.

Most likely releases to staff are automatic, though perhaps only once every 30 mins (as mentioned in the video). Release to production still require a human to say “go”, and that’d be a single click or a command somewhere.

Doubling up on releases (new)

On days that Facebook release to production, they are doing 2x what they were before. Chuck implemented this internally, then posted on public engineering blog. There was a backlash from members of the public that didn’t understand quite what he was referring to, and one commenter went as far as to accuse them of being staffed with ‘Glue huffing Monkeys’:

The middle branch

Facebookers request cherry picks to go to an integration branch on those few days that the new release branch is being production hardened. If they pass Jenkins (or equiv) various pipeline stages, there is an auto-deployment to a modestly scaled internal stack that Facebook staff hit when they attempt to go to ‘http://facebook.com’. As a result, they’re forced to sit in their potential you-know-what, though the official name for this practice is ‘dogfood’.

Mapping branching models to release frequency

There are certain branching models that suit a higher frequency of release to production. Trunk-Based Development is certainly more productive in that regard that the traditional ClearCase branching models. If you need to push to production more than Facebook are doing you’ll have to shift to a full Continuous Delivery model, and not do “make a branch for release” more than once ever. It could be you deploy from trunk (scripted as always), or that you’re merging to a permanent release branch (like with GitHub pull-requests).

As a consequence, you’ll see that the role QA in a team also disappears before you increase to daily releases. At the same time, automated functional tests become super important. Frequently companies jump to “too many” of them, from “too little” as they strive to bank the value from them.

Lots of companies have toggles in their production stack. Even the “release once a year” teams do. As you move to more frequent production pushes some of the toggles pertain to unreleased features, which is one place that traditional certification-centric thinkers get uncomfortable.

Lastly, released that are regretted: Some companies towards the left will have downtime as they work out what to do, or suffer the consequences of leaving that regretted release in production. Others will have the ability to downgrade a stack via scripts made for that purpose (per release) and practice that in staging. The likes of Facebook on the right design for rollback by having everything forwards and backwards compatible. Yes that includes DB changes as far as possible.

Not shown - how performance testing changes from left to right. It stops being a department and a gated stage to a release as the frequency moves towards the right.

New to Trunk based Development?

Granted Trunk-Based Development is not the pinnacle of continuous delivery, but for enterprises it could well be as far as they go. Read about Trunk-Based Development, Branch by Abstraction, and Feature Toggles:

Myself: Trunk, Branch by Abstraction, Toggles, Toggles under source-control Martin Fowler: Feature Branches (the perils of), Feature Toggles Jez Humble: Branch by Abstraction

Jez’s book is also a definitive tome on the subject.

Mar 15, 2013: This article was syndicated by DZone