For enterprise platforms that integrate multiple partners or clients onto a base platform, CMS, Rules, ETL, toggles, case management and BPM (count ‘em: six) all share a couple of common factors. Partner, client or customer config:

1. Needs to be deployed into production at a different release cadences than the base product itself.
2. Needs to be “developed” by a different team than the one that makes the application binary (platform or base product)

I would say that those six are all forms of configuration on top of the binary platform, as shown here:

I’ve three partners illustrated - Banana, Orange and Mango.

Promotion not Deployment

The trouble is that deployment isn’t quite the right word for what happens when the business decides to go live for one partner (but not others), it is “promotion” and it can be quite granular. The promotions activity is orthogonal to the application binary deployment cycle, and has a flow through environments towards production.

There is a general promotion problem once you decide this is what you need:

Consider a tiny JSON document as representative of any of CMS, Rules, ETL, toggles, case management or BPM as they pertain to a single partner/client. Changes over time (left to right) are outlined for a single partner below, as are the promotions through environments (each horizontal line):

Specifically, If we had a hot-fix in the middle of a long running change, we’d need a way to work on the former while not waiting for the latter to complete (because we need to fix something ASAP). Environment PS Dev1 is where a large team was taking many months to finish that large refresh of one partner. PS Dev2 is an environment not being used for such a involved change, and someone to quickly change the config, see how that behaves, and begin the promotions path towards production, without being entangled in a larger unfinished change. PS stands for “pro services”, by the way, which is an I.T. department designation for that team with a separate purpose (and funding model).

As I have it in the diagram above, the promotion to production was NOT re-typed in the PS-Dev1 environment and we’re due a regression when the long running change finally goes live. Hopefully that’s clear above - bad timing on promotions causes a regression. Incidentally, there are many subtle variations of the above that also cause regressions and lost work.

Formal merge is what’s needed

What you really need is to merge during a promotion, and there’s pretty much no technologies out there for those six that doesn’t simply overwrite during promotion. That’s a problem - regressions are embarrassing. Therefore, there is a desperate need for these types of things to keep their configuration as source as we conventionally know it, and under formal source-control, and use a formal merge behind the scenes during promotions. You get a ton of safety with that, though perhaps it is a bit harder to code for than today’s use of relational schemas. Note that’s not backup to source-control, that’s canonically store in source-control.

One of the key safety aspects is the ability to do a dry-run, where you’d get a detailed warning about a potential clash during a promotion, and time to remediate it.

While I show a general flow of promotions from Dev through UAT to production, it’s perfectly possible that you could promote from hot-fixes (and business-toggle flips) made directly in production, back to non-production places where they are needed for posterity. Indeed, in our particular case, a promotion of that one change (beef to chicken) could be made from Dev2 to Dev1, thereby preventing the latter error. Yes, people change config in production, if the need is urgent enough (or the config change is practiced enough - AKA safe).

For a proof of concept two years ago for a “business toggles” application called “App Config App”, Logan McGrath and I made a Perforce-backed system, that had promotions built in. Logan gives decent screen-shots of it in use in a blog posting. We had dry-run and meaningful promotion paths coded in that demo. Perforce was chosen because it has strong branch-spec concepts which aid promotions. There’s also a lesser evolved Git implementation there, and a started by not finished Subversion one.

Lastly, you’d want some of wall up between true development (where the application binaries are made), and this place where people are mostly configuring business entities (partners / clients). If there’s source control systems underpinning these it is different ones for Dev/QA and PS-Dev/UAT/Prod:

Note that diagram is modified from one used in the first in this article series: Application Architecture in the CD Era for Pro-Services teams (scroll down 2/3). The second in this series adds value too, perhaps: Provisioning, Deployment and Application Config Cycles