Paul Hammant's Blog: Trunk, CI, Builds, Environments, And integration
At a certain level of accomplishment a dev team has become adept at using build infrastructure to lock in their ‘Continuous Integration’ (CI) achievements. Martin Fowler pointed out “[…] It is only Continuous Integration if it’s run on a shared mainline that people commit to every day. Running such a daemon otherwise, such as on every FeatureBranch, is Daemonic Continuous Integration that debases the name.”. Thus CI is greatly overused as a term in the industry.
Nirvana is where teams are truly doing Continuous Deployment (CD) into production. They are not slowing themselves down with branches. Well, ignore the short lived feature branches of Pull Requests, that is.
For everyone else branches are a bigger factor. Here’s a typical modern branching model with overlaid build automation bits and pieces:
In the diagram above, the team has three usages of the build infrastructure. One is testing the work that is prior to integration into trunk or master. That is a commit/push as normal but to short-lived Pull Request branches. If the dev team is 20, and that work was from one dev, there are 19 that do not need to be alerted to the fact that one person committed and pushed something that broke the build. While the work on that PR branch should not go any further before it is fixed, that’s a fact that can be communicated in the web-UI for the pull-request workflow and not spammed to 19 developers that are busy with their own stuff.
Another is taking the very infrequent cherry-pick merges to release branches and running roughly the same build steps and technology, but with the additional step of deployment to an environment. I say that is UAT, as QA has already been done.
The third is the one that picks up commits/pushes to the trunk or master and runs them though the same builds steps. That one is the only one of those three builds that is Continuous Integration. The other two are not CI even if the infra to perform them is shared (Jenkins depicted in the diagram).
In terms of language “Integration” is the act of landing commits in trunk or master for all to see, and in a way that the humans involved have a high confidence that the commits are great. Specifically if the CI build fails then that should be a huge surprise, and have some random or bad-timing factor perhaps as the cause of that. There are many teams that hld expect something to fail in that consequential build, or won’t know until the next morning. Those are teams are not doing CI. And while “merge” on some basis is the tooling technique to land those commits, it is “integration” that is the useful descriptor for the action and intention.
Something worth reminding
In Trunk Based Development release branches are created ahead of the release(s) that they are pertinent to. They are not merely reflective of a release that has already gone out. For adept teams creation is a few days ahead, and for lower cadence teams as much as a week. As all branches, there is an underlying policy that drives the rationale (or should be). In this case we want a freeze, but not to the place the developers are most active (the trunk or master). We’re ensuring that the release is ready (production hardening) and that requires keeping developers out. They can fire hose into trunk/master if they like (one commit every few seconds in Google), but that release branch is a place where the list of commits making it into prod is stable. And when we decide that the branch contains a bug (can be discovered pre or post release), we fix it in trunk and cherry-pick it to the release branch. We do that because we do not want everything else that was fire hosed onto trunk/master to merge into the release branch too. Hence the ‘5&9’ label in the second release branch. Oh and release branches, like short-lived-feature branches get deleted after a while.
New build-infra frontiers
Many teams are able to use the build infrastructure tactically to verify partially complete work on a dev work station. This may or may not have been committed yet, but it certainly has not been pushed to any team observable place in the branch diagrams above. Specifically not to the pull-request branches (short-lived-feature branches). The use of build-infra could be Selenium nodes via “Selenium Hub”, but it could also be something closer to “the full build” if required. Specifically the changes are snapshotted somehow and handed to the build infra to run elsewhere while teh developer pushes ahead with what they are working on. I.e the news will come back “would or would not break the build if commit/pushed”. For a few years now Devs have been able to run technologies on their workstation that ran tests as they typed, but there are some build steps where that’s impractical so the “snapshot and run elsewhere” capability is needed
More on policies for (release) branch creation
“Branch: only when necessary, on incompatible policy, late, and instead of freeze”
— Laura Wingerd & Christopher Seiwald (1998’s High-Level SCM Best Practices white paper from Perforce)
The freeze aspect is key here. We do not want to freeze the trunk because that is where all developer’s commits are landing in quick succession (directly for v small teams, or more likely through pull-request mechanisms these days). We don’t even want to freeze it at any moment before a production release even temporarily. Instead we freeze the branch we make from the trunk. We do that with three mechanisms, when create the branch, we do not sweep commits since branch creation to the branch with a general merge. We also don’t admit the core of developers to that branch (in fact we lock out their ability to commit there), Lastly we have controls around cherry-picks (a specific type of merge) to the release branch because we only want genuine bug fixes there, and not “late” feature development (resist those business people and PMs!).
The short-lived feature branches uses are for temporary individual developer isolation from trunk, the pull-request and code review before merging into trunk? They’re different. Indeed, in Google in the mid 2000’s (perhaps 8,000 committers in one trunk) they were not branches at all, they were curated patch sets that facilitated many possibilities. This because Perforce did not have very lightweight branches.
The ‘late’ and ‘incompatible policy’ are correct. If you’re doing release branches make them at the last responsible moment before the release itself. If you need user sign-off for that release from UAT, then accommodate that. Do not make release branches early in each release cycle - though I’ve never heard that suggested by anyone who gets Trunk Based Development. The defining incompatible policy is that the release branch must tend towards stability (the trunk never will unless your org stops paying for developers and QAs).