I’m obsessed with SCM being the backing store behind apps of various types. Here is a guy using Git as an admittedly slow NoSQL database. I’m sure I’m going to use this someday, but maybe not for this particular idea.

Toggles on live apps.

Today I’m going to talk about configuration of web-apps, or any app that’s server-side that’s deployed as such. I’m talking about the XML, JSON or property file configuration for an app, that may be configurable before deployment to a particular environment and reconfigurable while the stack is up. For example, Feature Toggles. These could wrap any level of granularity of switch around functionality within the app. Of course flipping that toggle in a running machine should be a known and tested beast.

One example of a toggle designed for “hot reconfiguration” might be your employer’s bitcoin integration, and the fact that it could become non-responsive for your shopping cart app without notice. In that situation, you’d disable it from the shopping app’s UI (and all the click thoughts to it) and run with say ‘paypal only’ for a period of time. When the bitcoin partner service reactivates, or becomes accessible again, you’d toggle it on again.

A custom app for config management

Specifically, I wondering if there’s an app you can stand up per environment that allows live configuration of toggles (implicitly for that environment). That would save (and load) from a local-to-it branch of some configuration. There would be one branch per environment. Configuration would originate in a shared developer environment (or a fork thereof) and get merged in appropriate directions. From a central command-control place pulling of config from that deployment environment to a canonical store (still to/from a branch representative of that environment). Here’s a diagram representing that:

Those dotted lines are TCP/IP subnets, DMZs, fire-walled areas or whatnot. Only the central one can see the others. The config-app in each env can only (in the simples incarnation) load and save config from the designated folder. If the file changes, it reconfigures the values in the UI. If it changes in the UI (and the administrator hits ‘save’), it overwrites the document in the underlying directory. That directory is under the control of a SCM tool. The tool should never encounter a merge conflict in itself. From a central TCP/IP subnet, an administrator can push/pull the changes to/from the central SCM repository. File names are identical across environments; It is only the content that changes.

As well as the config-app, the actual business app is also pulling config from the directory in question. It wakes up and reconfigures itself whenever it’s locally changed. That’s a directory watcher, or a messaging technique to wake it up. The business app never writes to that directory though, it is always a slave of it.

Note - A couple of clients of mine over the last two months will recognize this diagram as I’ve proposed it to them, and talked it though as a way of separating code and config (and the release schedules thereof).

The administrator call also merge changes from one environment to another. They’d not merge all changes. They’d merge only the bits that are meaningfully the same. Other divergent changes would be merged AND overwritten before commit. Here is a the QA instance, with everything toggled on, and a new feature “cart size” that’s not gone to production yet:

{
     "env": "qa",
     "zipCodeValidator": "qa.zipVal.example.com",
     "zipCodeValidatorToggle": "ON",
     "bitcoinPayments": "bitcoinMock.example.com",
     "bitcoinPaymentsToggle": "ON",
     "upsellService": "upsell_suggester.example.com",
     "upsellToggle": "ON",
     "cartMaxSize": 10
}

Here is staging’s current configuration. Bitcoin is off here, as we’re as perhaps that scenario we suggested is could happen, has happened. The cartMaxSize element is not present at all in staging (yet):

{
     "env": "staging",
     "zipCodeValidator": "staging.zipVal.example.com",
     "zipCodeValidatorToggle": "ON",
     "bitcoinPayments": "staging.bitcoin-exchange.cc:23001,staging2.bitcoin-exchange.cc:23002",
     "bitcoinPaymentsToggle": "OFF"
}

There are several compelling aspects to this design:

1. A Custom UI can present meaningful affordances for elements of the underlying document. That would make it usable by non-developers
2. There’s still a directory and standards based documents underneath (XML, JSON, properties).
3. Sets of changes can be progressed together close to atomically.
4. Humans, on the command line, can arbitrate of complexities of merges for environment creation and config propagation.

For that last, it could be that on

Workflow via Constrained SCM Merges

Perforce (the SCM) has a feature called “branch-specs” which are neat. They allow you to set up approved merge mappings for, say, branches versus trunk. For our per-environment branch design, we could enforce SharedDev only be mergeable to/from QA that is only mergeable to/from Staging, that is only mergeable to/from Production. The thing you could stop with this, is a merge from SharedDev straight to Production without going through the right intermediates. Perforce also has permissions. Permissions are useful, as they will block a checkout of SharedDev’s config being upgraded to Production’s config (allowing nefarious changing of that config).

There’s no such thing as branch-specs in Git (or Subversion), so there’s no guard against inappropriate merges in this way outlined above. Similarly, there’s no fine-grained permissions on branches within a single repo for Git, so there are some problems with potential vulnerabilities. With help from Apache’s HTTP interface, Subversion too can support permissions on individual directories. TFS also has fine-grained permissions (of course it does, it’s a copy of Perforce :-P).

Perforce, TFS, and Subversion allow us to maintain divergence. With some arcane command line fu, so does Git. I’ll highlight the Perforce way though as I explain the importance. Look above in the code snippets for ‘zipCodeValidator’. There’s some clear machine-centric values there that you’d never want to merge from one branch to another, as they are ALWAYS going to be different. It does not matter whether they are created correctly at original branch creation, or created as keys later. You mostly mask them out of merges of config. In Subversion we would do a merge utilizing “–record-only”. For Perforce it is an “accept yours” option during a merge conflict resolution. Basically, after these options the merge is not offered again, even though the difference is stark. The trick is that you have to merge back in the opposite direction in a similar way too, given we’re seeing app-config as a workflow through a fixed series of SCM branches now (as shown above). Directions of workflow are thus situational.

The managed divergence born from –record-only or “accept yours” is one of the key aspects to this proposal.

In terms of tooling, the app could have a dashboard like so (complete with button to initiate the promotion of a set of changes):

Power users may prefer the command line for merges. That or they may be too complicated for the UI to perform. I’ve a love for “round-trip” modification of things. In this case a custom UI, but also possible in lowest common denominator tools.

Other notes.

The config app is not coded to know about individual configurable items. Instead it has a set of validations that are enshrined in a document of sorts:

[
    {
        "name": "zipCodeValidator",
        "type": "DOMAIN_PORT",
        "default_port" : 12233
    },
    {
        "name": "zipCodeValidatorToggle",
        "type": "RADIO",
        "values" : [ "ON", "OFF" ]
    }
]

Thus it does not really matter what the servers-side of the app is coded in. It should be no surprise to hear me say that you will most likely use AngularJS or Knockout to code the client.

Puppet, Chef, Further reading.

Puppet works just fine with SCM as it’s canonical store. It is an evolved tool chain for provisioning, rather than something that’s going to facilitate hot-reconfiguration of an app, that probably uses custom loading as we outline. That does not mean that you cannot co-mingle Puppet/Chef and app-stack config in the same SCM branches. Indeed, they are perfect bedfellows and the same merge metaphors work well.

Colleague Max Lincoln points out that Puppet for one has an evolved set of practices for config across environments, and that there is a blog article talking through the pros and cons of various configurations. He also directs me to Hiera that can do hierarchical setups of config data. I’m not so much interested in that because I’m enamored with the merge capabilities of a good SCM backing store. Similarly I’m not so much interested in inherited config, however that is achieved.

A Recent ThoughtWorks developer conversation talked through Twelve Factor’s app-config advice and that is all pertinent to this blog entry + proposal. Well maybe not the must-be environmental variables stuff.

One more thought on Git.

Some crazy dude has hooked Git up to Solr to explore history. Exciting times.

Update 14th Aug 2012

Here is (the start of) an implementation using Git/GitHub