Hi Paul,

Since you recommend option B, I'm curious to hear your recommendations on how to tackle the associated challenges...

1. The “Diamond Dependency” Problem - Were you thinking that you'd have more than one pipeline? If you have a single pipeline you'd find out pretty quickly if someone introduced a dependency problem, right?

2. Having a supported workflow for developers changing code that’s not in “their” module - Assuming you haven't set up access control to restrict what people work on, would you rely on the first testing phase of the pipeline to catch any inadvertent commits to other modules?

3. Making sure changes to shared modules are fully tested against tens or hundreds of applications that use them before committing. - Sounds like a good place for pre-flight builds. Otherwise you might pass the commit phase testing and break a lot of downstream phases.

4. Making sure changes to shared modules are in line with the directions of the 10s/100s of apps that use them. - I guess you'd want enough acceptance testing that you can catch errant behavior even if it doesn't directly violate an existing requirement.

5. Worrying about the size of representation of the ‘checkout’ on the developer workstation, maybe needing tooling to reduce that. - I've seen a couple of good tricks in this area. One is using more selective SCM checkouts to narrow the data given to the user, and the other is relying on network storage for working copy management. The latter approach works pretty well for projects with over a few GB of data included.

6. Worrying about the update/sync/pull/fetch duration being a disincentive to doing it 20 times a day, needing tooling to reduce that. - Do you mean in the sense that it just takes a long time to figure out what's updated, or perhaps that there's a lag in transmitting the updates?

1) In terms of CI, yes you'd have pipelines for all deployable sets of binaries. Catching the problem would be quick, arbitrating between 'team coco', and 'team america' one whether "we" should be using Hibernate 4.3 or 3.2 is harder.

2) wouldn't set up access control as you hint, and yes to CI catching things.

3) yes.

4) yes, and forums to discuss directions for shared tools (committees, mail-lists etc).

5) Yup, your source-control tool Perforce does that well - client-spec (Subversion has an approximation).... I hate to say it but ClearCase can do it too.

6) relates to 5 perhaps. Only "sync" what you're about to build.

For Google I would recommend reading http://google-engtools.blog...
I will be also talking a bit more about branching at http://releng.polymtl.ca/ -- basically "B".

yay!

Nice post. Ironically, you wrote this post several years ago and it is still relevant today. The company I work for follows a pattern much like your B pattern. Surprisingly, this is one of the few blog posts I've come across where such a pattern is described in detail.

Ironically, however, the second "benefit" you describe for that pattern, namely "Continuous Integration that reports defects with seconds of them being committed to the trunk.", seems to be unachievable in certain cases including ours. Perhaps you could put that benefit within a context that explain how you have been able to achieve such fast turn around times at scale.

See, we are primarily a C++ shot, and our codebase has close to a dozen different applications built upon several common "framework" layers, all managed from a single trunk. On high-end hardware (i7 CPUs (8 core, 3.4Ghz+), 16-32GB RAM, SSDs, GigE networks, parallel build operations, etc.) it still takes in excess of 8 or 9 hours to checkout, build, test and package the entire codebase from a clean for both 32bit and 64bit platforms, and debug and release configurations. Of course incremental builds are somewhat more reasonable, often being completed in under an hour, but we have not yet found any way to reduce the turnaround times to the degree you are claiming. Not only does this pose problems for CI and automation, but it presents unique difficulties for developers on their local builds as well. In the degenerate case, it takes so long for a developer to compile and test their code changes locally that by the time they complete that procedure someone else has committed more changes that they then need to pull locally before committing. But then they need to repeat the whole process again to make sure the new changes they've just pulled in haven't introduced any new problems, and so the cycle repeats.

I think the issues we are experiencing may be particular to the tooling. You refer to Java in your article, for example, and mention how Maven takes care of managing the dependencies and generating build graphs automatically. I believe .NET has similar tooling as well. We are currently a Microsoft shop and unfortunately the Visual Studio tools don't scale well in that regard (try loading a single solution file with thousands of projects in it and you'll see what I mean), and unfortunately any non-Microsoft tools that may be able to help in this regard typically don't play nice with the Visual Studio toolset.

I'd love to hear your thoughts on this and to see whether you've heard of anyone successfully applying this pattern in a production environment at scale. Unfortunately for us, it seems the only "practical" long term solution may be to consider patterns similar to your options C and D, and then looking for ways to manage the large number of interdependent modules individually. However, once again I have not yet found any off-the-shelf tools that are compatible with C++ and Visual Studio capable of doing that so we'd be faced with hand-rolling a solution for that as well. :(

Lots of Games studios use "one big trunk" (and perforce in particular). Frank Compagner of Gorilla Games talks that it in various blog entries and videos. Even if you have 1000 Jenkins slaves to do builds in parallel, its still a 8-hr turn around on good/bad news.

No. Your answer is to do what Google do. Their fabled build system "Blaze" was open sourced as Bazel. C++ is their heavy-lifting language and is even used from many Java solutions. ExGooglers at facebook made "Buck" in Blazes image (before Bazel was open sourced).

These two (three) build systems allow you to build a direct graph of modules. The considerable CI infra that is tied to them allows them to focus at the directory level and build linkable objects per-directory. That way it's easy to see what has changed versus the last build (even for huge trees) and rebuild only the bits that have changed. Similarly, impacted tests can be determined and only those are run. You might be able to ring the per-commit CI-driven build/tests cycle down to minutes. Maybe, in a different CI loop, you still build "clean" back to back, as a safety net on the possibility that the incremental build might have false positives.

If you have too much source for Git to handle, consider Perforce. Even Microsoft used it back in the day - recompiled for them as "source depot" (SD).

The main deterrent for us in adopting tools like Bazel and Buck is that they don't interop with Visual Studio. Make, Gradle, Scons and pretty much all the other similar build tools I've evaluated have the same limitation. At best the tools allow you to generate Visual Studio projects from their own custom, proprietary configuration files. This leads to maintenance issues because developers are accustomed to making configuration changes directly in the IDE, and those changes don't automatically get reflected back to the config file the third party tool used to generate the project. I mean, short of using the Visual Studio IDE as a glorified text editor and forcing developers to maintain the build configuration using text files, I'm not sure how that would be maintainable.

Do you know of any tool or development shop that has solved this particular problem? For example, is there any way to setup Visual Studio such that any project changes that are made via the IDE are reflected somehow back to the source files that generated the project without manual intervention by the developer?

In the early days of Maven, the worst-best-IDE Intellij didn't support it in the UI. So you were forced to edit pom.xml files in the IDE as a "glorified text editor" :). The flipping to the command line to run a build. Maven has a recursive build. From root, for you that would still be 8 hours. but maven would allow you to cd all the way to a leaf module and build/test that. It achieves the same goals as Blaze, but implements differently. For Blaze (etc), you remain in the root dir for the project and pass the sub-module as a parameter when you wanted that focussed build/test. You have to have done a build test at least once of course for the linking to work seamlessly. Now Google's other genius for that was that they put previously built permutations in an LRU cache for the whole team (and CI to depend on). Say you have a directory (a module) in a tree with 30 C/C++ files in. Make a hash of the contents of all of the sources within, have that as they key, the payload being the intermediate linkable object for the same completed compile artifact. Ties you all to the same V.e.r.s.i.o.n of the compiler but that's not a bad thing. I can't remember whether Googles LRU cache backed by memcached or not (I left in Jan 2009).

To me - shortening your build times trumps developer IDE comforts. Sure - without a plugin for the IDE (Google made that too), you are dependent on the command line, and the ability to attach a remote debugger to that process (say you're breakpointing a failing test).