Update: See the new resource site for Trunk-Based Development called, err, TrunkBasedDevelopment.com and make sure to tell your colleagues about it and this high-throughput branching model.

I’ve been pushing this branching model for something like 14 years now. It’s nice to see Facebook say a little more about their Trunk-Based Development. Of course they’re not doing it because they read anything I wrote, as the practice isn’t mine, it’s been hanging around in the industry for many years, but always as bridesmaid so to speak.

If not Trunk, what?

Mainline?

Mainline as popularized by ClearCase is what we’re trying to kill. At least historically. It’s very different to Trunk-Based Development, and even having vastly improved merge tools doesn’t make it better - you still risk regressions, and huge nerves around ordering of releases.

ClearCase’s best-practices also foisted a ‘many repos’ (VOBs) on teams using it, and that courted the whole Conway’s law prophesy. I mentioned Conway’s Law before in Scaling Trunk-Based Development and it concerns undue self-importance of teams around arbitrary separations.

Multiple small repos for a DVCS?

There is a great statement by a Reddit user in the programming section of Reddit, in conjunction with the Facebook announcement:

Comment ref or all comments

This Redditor is right, there’s a lack of atomicity around a many-repos design, that stymies bisect. It could be that Git subtrees (not submodules) are a way of getting that back (thanks @chris_stevenson on a back channel). There’s also a real problem moving code easily between repos (with history) though @offbytwo (back channel again) points out that subtrees carefully used can help do that.

Trunk at Google vs Facebook

Tuesday’s announcement was from Facebook, and to give some balance, there’s deeper info on Google’s trunk design in: Google’s Scaled Trunk-Based Development.

Subsetting the trunk for checkouts

TL;DR: different

Google have many thousands of buildable and deployable things, which have very different release schedules. Facebook don’t as they substantially have the PHP web-app, and apps for iOS and Android in different repos. Well at least the main PHP web-app is in the Mercurial Trunk they talked about on Tuesday. I’m not sure how the iOS and Android apps are managed, but at least the Android one is outside the main trunk.

Google subset their trunk. I posted about that on Monday. In that article I pointed out that the checkout can grow (or shrink) depending on the nature of the change being undertaken. It’s very different to a multiple-small-repos design.

Facebook don’t subset their trunk on checkout, as they do not need to; The head revisions of everything in that trunk are not big enough for a C: drive or IDE to buckle. There’s also no compile stage for PHP, for regular development work.

Maximized Sharing of Code

TL;DR: the same

Code is shared using globbed directories within the source tree. It’s shared as source files, in situ, rather than classes in a jar (or equivalent).

Refactoring

TL;DR: the same

Developers take on refactorings where appropriate. Sure it means a bigger atomic commit, but knowing all the affected source is in front of you as you do the refactoring is comforting. At least, knowing that if Intellij (or Eclipse, etc) completes the refactoring there’s a very strong possibility that the build will stay green, and that you’re only going to have a slight impact on other people’s working copy, and only if they are concurrently editing the same files. Bigger refactoring probably still require a warning email.

Super tooling of the build phase

TL;DR: the same

Google have what amounts to a super-computer doing the compilation for them (all languages that are compiled). All developers and all CI daemons leverage it. And by effective super-computer, I mean previous-compiled bits and pieces are pulled out of an internal cloud-map-thing for source permutations that have been compiled before. The distributed hashmap is possibly LRU centric rather that everything forever.

Facebook don’t have that big hashmap of recently compiled bits and pieces, but they do have HipHop in the toolchain (originally a PHP to C++ compiler) which is interesting because, at face value, PHP is an interpreted language and ‘compile’ makes no sense. HipHop was created to reduce the server footprint and requirements for production deployments, while still being 100% functionally identical to interpreted PHP. It’s also faster in production. More recently HipHop became a virtual machine. It continues to be incrementally improved. Like Google, Facebook can measure cost-benefit of continued work on it (prod rack space & prod electricity vs developer salaries).

Source-Control weapons of choice

TL;DR: different

Google use Perforce for their trunk (with additional tooling), and many (but not all) developers use Git on their local workstation to gain local-branching with an inhouse developed bridge for interop with Perforce. Facebook uses Mercurial with additional tooling for the central server/repo. It’s unclear whether developers, by habit, exist with the Mercurial client software or use Git which can interop with Mercurial backends. Both Google and Facebook do Trunk-Based Development of course.

Branches & Merge Pain

TL;DR: the same

They don’t have merge pain, because as a rule developers are not merging to/from branches. At least up to the central repo’s server they are not. On workstations, developers may be merging to/from local branches, and rebasing when the push something that’s “done” back to the central repo.

Release engineers might cherry-pick defect fixes from time to time, but regular developers are not merging (you should not count to-working-copy merges)

Eating Own Dog-food

TL;DR: mostly different

All staff at Facebook use a not-live-yet version of the web-app for all of their communication, documentation, management etc. If there’s a bug everyone feels it - though Selenium2 functional tests and zillions of unit-tests guard against that happening too often.

Google has too many different apps for the team making each to be said to be a daily user of it. For example the AdSense developer may use a dog-food version of Gmail, but they are making AdSense, so are hardly hurting themselves as they are not minute by minute using the interface as part of their regular existence at Google.

Code Review

TL;DR: same

Both Google and Facebook insist on code reviews before the commit is accepted into the remote repo’s trunk for all others to use. There’s no mechanism of code review that’s more efficient or effective.

Google back in 2009 were pivoting incoming changes to the trunk around the code-review process managed by Mondrian. I wrote about that in “Continuous Review #1” in December. I think they are unchanged in that respect: Developers actively push their commit after a code review has been completed.

Facebook have just flipped to Mercurial (from Subversion). In the article linked to at the top of the page, Facebook have not mentioned “pull request” or “patch queue”, or indeed “code review”. The article was mostly about speed, robustness and scale. -I suspect they are sitting within the semantics of Mercurials patch-queue processing though, although assigning a bot to it rather than a human.- Update: Simon Stewart pinged me and reminded me that they use (and made) Phabricator. He spoke about it in a Mobile@Scale presentation, and that video is here. In the video he says the review is queue based now, but that they experimenting with landing the change sets into the master now. The video is from November, and was for the Android + iOS platforms, but it is likely to be used today for the main trunk for the PHP web-app.

Automated Testing

TL;DR: same

Heavy reliance on unit tests (not necessarily made in a TDD style). Later in an build pipeline, Selenium2 tests (for web-apps at least) kick in to guard the functional quality of deployed app.

Manual QA

TL;DR: mostly the same

Both companies have progressively moved way from manual QA and dedicated testing professionals, towards developers testing their own stuff at discrete moments (note the Dog-food item above too).

Prod Release Frequency

TL;DR: it varies.

Facebook for the main web app, are twice a day presently (at least on weekdays). I published info on that at the start of last year. Google have many apps with different release schedules, and some are “many times a day”, while others are “planned releases every few weeks”. Many are in between.

Prod DB deployment

TL;DR: mostly the same

Database (or equivalent) table shapes (or equivalent) are designed to be forwards/backwards compatible as far as possible.

Pull Requests as part of Workflow

TL;DR: same

Etsy, Github, and other high throughput organizations are trunking by some definition, but using pull-requests to merge in things being done. It has different obligations if done, but Google and Facebook are not doing this in their trunks - they both essentially push (after review). Refer the ‘Code Review’ section above.

Common Code Ownership

TL;DR: The same

You can commit to any part of the source tree, provided it passed a fair code review. Notional owners of directories within the source tree take a boy-scout pledge to do their best with unsolicited incoming change-lists. There are strong permissions in the Google Perforce implementation, but the pledge means that contributions are not often rejected if the merit is there.

Build Is Ever Broken

TL;DR: The same

Almost Never.

Directionality of merge for prod bug fixes

TL;DR: The same

Trunk receives the defect fix, it gets cherry picked to the release branch. The release branch might have been made from a tag, if it didn’t exist before.

Binary Dependencies

TL;DR: The same

Checked into Source-Control without version suffixing (harmonized versions across all apps). E.g. - log4j.jar rather than log4j-1.2.8.jar.



Comments formerly in Disqus, but exported and mounted statically ...


Fri, 10 Jan 2014Ludovic Urbain

Source control is a problem, not a solution.

Learn to develop without source control, and only then will you see the light ;)

100% of the reasons you use source control are unrelated to versioning, so why not solve your problem instead of adding another one ?

Fri, 10 Jan 2014Geert Theys

I'm not sure if you're trolling or not? But I'll bite: Care to explain what SC is THE problem?

Fri, 10 Jan 2014Ludovic Urbain

All form of Source Control is an overhead with the sole purpose to help manage avoidable mistakes through an inherently complex mechanism that brings its own problems along.

This, rather than fixing the original problems, which are bad application design and low developer skill.

Fri, 10 Jan 2014wtpayne

I guess you could use unison for commitless synchronisation with a "master" repo. on a central fileserver -- and have a VCS (git, perhaps) watching master backing up deltas for posterity.

Fri, 10 Jan 2014Ludovic Urbain

I'm going to have to ask:

Why do you need those versions anyway ?

What do you do with them once they are outdated and never run anywhere anymore ?

What is the point of storing deltas when you only need the final working version anyway ?

Fri, 10 Jan 2014Billy O'Neal

Because finding out who made a change on a given line of source, at what time, is an extremely useful question that VCS can answer for you. Also, often you need more than the final working version; e.g. if you need to make a showstopper fix to a released version after the development team has moved on to the next version.

Sat, 11 Jan 2014Ludovic Urbain

1. Wrong. Nobody cares about changes that do not cause problems.

2. Why do you need another version ? Why do you have more than one version ?

Sat, 11 Jan 2014Billy O'Neal

1. Wrong. If you are the first person looking at a codebase in years, being able to find where the bodies are buried is extremely useful.
2. At a minimum, you have the last released copy, and the working copy. Presumably you are not shipping to customers every time you hit save in your editor.

Sat, 11 Jan 2014Ludovic Urbain

1. I don't see how VCS are going to help you look at a codebase. I personally use notepad++ / vi / grep depending on where I am. But feel free to use VCS to read text files, I mean, why not.

2. There is Development and Production. Between commits, Development will diverge, and as it is pushed to Production at least once a day and as the Production version doesn't exist anywhere else than on the production servers, there is no concept of version or version control.

Thu, 30 Jan 2014An0nym0usC0ward

So you never heard of staging or canary deployments?

Fri, 31 Jan 2014Ludovic Urbain

It's a bad idea. Don't do it. You don't want the entropy.

Do your testing properly, and such approaches will never make sense.

Tue, 15 Mar 2016An0nym0usC0ward

You do want a certain amount of entropy.

A codebase or a team not being able to deal with entropy is fragile - the first significant perturbation to come along will cause a catastrophy.

The only way to learn to deal with entropy is exercising. Look at what Netflix' Simian Army does to their products.

Fri, 10 Jan 2014wtpayne

First of all: The Unison suggestion was not even slightly serious. Git, Hg, Svn, P4: all much better options by far.

Secondly: I write machine learning algorithms for a living. It is frequently necessary to re-run old algorithms against new test-data (or newly ground-truth'd test-data) to prove that your performance improvements are actually improvements and not regressions. Being able to reproduce old results is a cornerstone of maintaining credibility with our customers.

Even without the need to re-baseline performance measurements, I still have an occasional need to go back through the source code history to determine when changes were made and why. This need manifests itself maybe 3-4 times a year on average. (More frequently in a large team; less frequently in a small team).

By coincidence, earlier this afternoon one of my colleagues asked me why some code had gone missing. Using git log --follow --patch -- <filename>, we were able to track down when the change was made, who made the change, and (eventually) the reason why. Without this ability, we would probably have assumed that the deletion was accidental, and re-inserted the code, kicking off an unnecessary and completely avoidable edit war.

Finally, using a version control system is the functional equivalent of keeping a good-old-fashioned day book, recording what you did during the day and why. Are you ever going to go back and re-read your day book? The chances are, no, but it cultivates a culture of deliberate action and professional discipline, as well as providing a record to back you up if you are ever involved in any legal disputes.

Sat, 11 Jan 2014Ludovic Urbain

Why do you start your secondly with an argument you know is flawed ?

The problem that made you use your version control search is a problem of process.

"Some code" just does not go missing without critical problems, in a well-designed system.

"Some code" just does not get deleted by the person not responsible for it, in a well-defined responsibility organization.

"Some code" just does not concern anyone else than that person, in a well-designed system.

I know dividing up programming work is hell, and it probably makes total sense for you to use VC to compensate for the imperfections in your design and processes.

I'm just saying I'd like to hear one good reason why good programmers should use version control.

Sat, 11 Jan 2014Billy O'Neal

You clearly have not worked on a system with many people. When programmer A leaves the team, and programmer B shows up, there is no process under the sun that lets programmer B magically get this kind of information.

Fri, 10 Jan 2014Richard Stanford

Its funny - I've been at shops with simple systems, complex systems, big teams and small...

My ideal setup actually works well with a small team (could be a few people per product on many interrelated products) just using boring VCS (SVN etc are fine). We all worked in trunk and had regular weekly releases. Release code got branched off for posterity twice, once as a "release" tag and once as a LIVE branch in case a patch was needed (which would also get copied into a "release" tag). Features were broken down into sub-week chunks (often building interfaces and libraries first, etc).

If anything came up mid-week that would necessitate a branch, it either waited until Monday or we literally tossed whatever we had been working on away and rewritten the next week - or more often 6 weeks later when priorities had settled again.

The problem that we solved was that a lot of times in-process features would be "saved" through VCS branches, but the cost of switching states, merging half-done code back into the system 6-8 small releases later (is it still the "right" fix?), etc, actually outweighed the relatively small cost of the rare cases when something couldn't wait for an extra half week before we got to it.

Also, don't underestimate the value of a moronically simple code/review/build process. It doesn't solve the complex problems that a "modern" approach does, but it does make a lot of them just disappear completely.

Sat, 11 Jan 2014Ludovic Urbain

Well, the thing is, VCS, OOP, and many other concepts exist solely to enable bad programmers to deliver a working result.

So yes, simple is always better when you think simple, i.e. have great people make great stuff.

Sat, 11 Jan 2014paul_hammant

You seem to have some larger message here Ludovic. Not just on the usefulness of the trunk stuff I am pushing, but also on tangential things like OOP. Can you put something up on www.ludovicurbain.com please and link to it from here, rather than continue to roll more and more thoughts into the comments hanging off this blog entry? Thanks.

Mon, 20 Jan 2014David M. Chess

You're saying that perfect programmers would not need all this stuff. Probably true! But not all that interesting?

Fri, 24 Jan 2014Ludovic Urbain

I'm just saying VCS may not be the right answer to those problems, especially since it's very indirect, requiring a lot more energy due to that indirection, whereas fixing the real problems would probably cost less and cause less problems.

Really, just like OOP, it does help a ton in making average programmers deliver, but is it really the best direction, considering all the other problems it creates ?

I'd definitely like to see more questioning and focus on the real issues rather than one of the solutions that have been used to more or less address them over the years.

Sat, 25 Jan 2014David M. Chess

So what are these real issues that you think there are better solutions to than OOP and VCS? Those are (among) the best solutions we've come up with so far. You have something better? :)

Mon, 27 Jan 2014Ludovic Urbain

My point is simply that OOP and VCS are a fitting compromise for people willing to hire average workers, and should be presented as such.

Another possible solution to solve the same problems is to hire only the best, use FP (!strict) for code quality and manageability, build each level as a rock-solid foundation, and succeed by design rather than luck.

Mon, 27 Jan 2014David M. Chess

Well, my own experience suggests that that is incorrect: even the best developers do better with OO and code control, on any nontrivial project, in any language. But if you are right, then you are in a position to gain fame and fortune demonstrating it. I await the headlines... :)

Tue, 28 Jan 2014Ludovic Urbain

Yup, experiences differ.

Thu, 30 Jan 2014An0nym0usC0ward

Basically, you're arguing we should still live in caves, as hunter-gatherers, because technology, in any form, is just a crux for not-fit-for-survival individuals, right? Why did we bother to develop a brain, then?

OOP, at least partly, emerged when industry projects done in plain C became so big that you simply ran out of good names for functions and global variables. With encapsulation and information hiding you solve this problem.

In the beginning, VCSs didn't exist, and people were simply exchanging patches. This worked well for small codebases and reasonable numbers of patches, but when your codebase gets to millions of LOCs, and you need to juggle thousands of patches from one production build to the next, manual patch management is no longer an option. VCSs are just a tool for such needs.

As for simple, you try make the Linux kernel simple. In an imaginary, dictatorial world, there would be just one disk interface standard, just one video card type, a single processor manufactuer building a single processor type, and of course a single volatile memory brand - with all systems using the exact amount of memory, so as not to create the need of different memory manager implementations for different hardware configurations. And no other type of device or component whatsoever. Which would of course simplify the kernel a lot - for the single, standard OS everybody would use, on phones, tablets, notebooks, desktops, servers, routers, switches, TVs, controllers and any other device containing a processor alike. Go ahead, manage your code just for such an ideal system, I don't mind.

Or are you just trolling?

Fri, 31 Jan 2014Ludovic Urbain

My dear anonymous coward, OOP emerged when the industry wanted everyone to be a coder, not just smart people with a passion for machines anymore.

OOP does not solve the problem, it makes it worse by making a dozen obscure boxes instead of one large dumb box (C). That's still so much worse than using FP, reducing state to its smallest expression, and having vision over 100% of your project, and not just some collection of obscure boxes.

Patches was old, VCS is old today, I have automated version coherency and I don't need your crappy tools.

The Linux Kernel could be made simpler if it wasn't written in plain C, but rather in a dialect that allowed for pre-compile expansion, thereby limiting the source size and inherent complexity.

But since people struggle with basic things like making mdadm raid10 perform normally, I'm not going to be asking for that right now.

I'm not telling you VCS is bad because it's new, I'm telling you the whole concept is outdated and people should just stop bowing to it when it's obvious it's broken.

Wake up, realize it's broken, fix it. Stop talking about it, stop trying to make it better, transcend it.

Fri, 10 Jan 2014paul_hammant

<grin/>

The intentional Software people (http://www.intentsoft.com) don't have source as we know it, in their science. Nor source-control as we know it :)

Thu, 30 Jan 2014An0nym0usC0ward

How would you get around source control, in a form or another, when several devs work on the same code base? How about maintaining different branches for different customers?

Thu, 30 Jan 2014Ludovic Urbain

1) Not have the several devs work on the actual same code base. If you're editing one function and your friend is too, you're doing it wrong yes ?
2) Why would you do that ? Do you like headaches ?

Thu, 30 Jan 2014An0nym0usC0ward

Have you ever practiced what you're preaching? I mean, on a non-trivial, non-hobby project. I did, some 20+ years ago, and it's awful, compared to using a version control system.

And no, I don't like headaches - which is why I use version control systems. And I like paying customers, and paying customers like customized product versions (at least in the real world) - and are willing to pay for the overhead.

Anyway, good luck finding a paying customer with some money to spend which will give you work if you tell him you won't use a version control system.

Fri, 31 Jan 2014Ludovic Urbain

You're not getting it are you ?

You don't need different versions to serve different customers, you need one core framework, and one specific application code for that customer, no VCS.

Apparently, those who meet me realize that the quality I provide is much more important than the use of your beginner's tools ;)

Fri, 10 Jan 2014Max Kirillov

This does not explain, how do they manage to have the "almost never" failing builds. As far as I know about it, the only way to achieve it is to be able to pass the codebase together with proposed change to the same build pipeline which is used for production builds. This is trivially possible with a DVCS, but quite harder for "centralized VCS with a review tool" (but maybe they have implemented it).

Thu, 30 Jan 2014Amine Chadly

I think you can look into the huge review and pre-commit artillery that they put in place (briefly explained in one of the section of this article (https://paulhammant.com/2013... and that you can pipe in the chromium organization and development tools.

Fri, 31 Jan 2014Max Kirillov

> As you ‘pre-submit’ a Perforce change list for review, a daemon side of Mondrian kicks in and takes your pending change list and runs it
through multiple tests, adding more data to the Mondrian code review for that change. This is all available to the developer before they go as far to submit something for review, which is good as most people like to fail in private.

This looks very much like the "Short Lived Feature Branches" as classified in https://paulhammant.com/2013... , but the branches are simulated rather than got used natively from VCS. And this is exactly what I meant in my comment - to get stable trunk one should use some branches, at least short lived ones.

Using the simulated branches could make sense in 2006, because there was few VCS-es with good enough branching, but now mankind has advanced quite noticeably, so it can be just VCS branches with no additional tools like "review storage" or "shelves" or however it named.

Thu, 06 Feb 2014wtpayne

We use Gerrit, which is (AFAIK) the open-source grandchild of Mondrian, and yes, if your patchsets sit in review for any appreciable length of time (more than a few minutes), it does indeed start to look a lot like the short-lived-feature-branches approach that Paul Hammant describes. If somebody's Gerrit build is failing, I will sometimes fetch it, fix the bug, and submit a patch back to Gerrit. This probably happens once every couple of weeks. I am not sure that we have yet used pending Gerrit reviews as areas for deliberate, planned collaboration ... but it will probably happen sooner or later.

Fri, 07 Feb 2014Amine Chadly

@maxkirillov:disqus: I think that developers use scms locally, but that those branches remains out of the corporate radar. I think they do so for two main reasons:

- It probably makes easier the corporate scm administration (keeping it responsive when you have a massive code line is probably nothing short from a nightmare and I guess they feel that branches are just dead weight).

- it may make the code changes easier to test (specially in a multi-platform context) with smaller chunks to consider.
- dichotomies can be performed in the main line (this may be helpful for organizations that have a huge number of automated tests at their disposal and cannot run all of them on all commits).

@wtpayne:disqus : you might be interrested in the following (https://github.com/vadims/g... ) once you want to get into planned collaboration ;)

Thanks to you both for your answers.

Fri, 07 Feb 2014wtpayne

Amine Chadly - Thanks for the heads-up on the Gerrit plugin.

I was talking about this with one of my colleagues earlier today, and it looks as if we might use secondary repositories to support pre-review collaboration.

I should mention though that I am pretty wedded to the C.I. model of development, so I don't really like anything that distracts people from a focus on doing tiny bite-sized bits of work and integrating very very frequently. (A pre-review "throwaway" integration testing repository might be OK, though).

Mon, 10 Feb 2014Amine Chadly

Thanks for the great conversation ;)
I think additional repositories is too much overhead (admin work + merge pain).
I lean towards strong code ownership, pre-commit testing + optional review ;)
But anything that works for you is just fine !^^!

Thu, 30 Jan 2014An0nym0usC0ward

I can't recall where I read it, but there's a blog post somewhere on the Internet documenting at least Google's process. Basically, when you attempt to commit code, a snapshot of the repo is taken, and all tests in that snapshot that might be affected by your changes are run before your code is actually committed - which tests those are is determined using dependency analysis.

This doesn't necessarily mean bugs don't make it to production, it just means that builds are very unlikely to fail - in fact, they can fail only when two commits happen concurrently and the changes they introduce collaborate to break the build, or when the dependency analysis misses some tests, and those do break due to a change being committed. Both of these situations are highly unprobable, provided commits happen often and don't take long to finish.

Thu, 27 Oct 2016Luís Alberto Costa

Very cool that matter, actually generates much discussion, they know where I can find more about it?

If you think 100% of the reasons that you use source control are not related to version control, so why not solve the problem instead of adding another?