The title is quote by Liam Holohan. Clever lad.

My article yesterday outlined a blueprint grammar that would be held in source-control for easy environment provisioning and de-provisioning. The theory was dev teams might relinquish environments if they could get them back more easily, and you would not need as much approvals red-tape for them. The TCO of running environments would become cheaper by a good deal.

I avoided discussion of “prod” in yesterday’s article, as you don’t casually drop and recreate prod when the blueprint changes. You could, however, easily use the same technologies for the initial creation of prod in your virtualized hosting facility.

Practicing Disaster Recovery

You’d make a branch in source-control of course, and ripple through all the domains changing to some transient domain like petstore.projectfoobar12.com. At least, for all the applications you’re about to practice with. Before you start, you’d make sure your virtual hosting provider has enough capacity for the experiment, and the corporate credit card can flex to cover it too. Then you’ll pull the trigger and provision everything needed to make production-like environment, deploy the actual app to it, after double checking you’re not hooked up to the production database, but only to the new one that’s for this experiment. This is easier with open source databases, than it is with Oracle or other fee bearing ones (unless you have an all-you-can-eat license).

After provisioning, and recording the elapsed time for the experienc report that your writing, you’d attempt to load the stack to see how it performs. That perhaps means an online testing service like Welcome to Neustar Web Performance Management (used to be called BrowserMob which was snappier), which will also impact the corporate credit card.

Finally, you’ll:

• de-provision the whole stack and add the accumulated dollar charge to the report you’re writing
• delete the branch in source-control

Note: I’ve written before around the ‘practice’ aspect of this - Continuous Delivery: Professionals vs. Amateurs

Practicing and benchmarking more than one cloud for your app

You could easily repeat this test with more than one cloud provider. There are new ones coming to the market every year. We could easily count hundreds by the end of the decade, even if some consolidation could happen later. It seems to me that the cost is never going to be crazy for a time-boxed experiment, so you really would want try more than one. A Forbes contributor thinks there will be a large collapse in the cloud provider market, but I’m not convinced nor concerned by that, unless prices go up. VMWare’s software makes it easy to get into that market, and perhaps they even help startups do so [didn’t turn out to be true: 2022]

Actual Disaster Recovery

Once you’ve banked all the practice, you’re in a position to use it when it’s really needed. It could be that your main provider has gone off-line. Bringing up an alternate, even if the performance is not quite as good, should be procedural now. That is, with caveats:

• You’ll have to reconnect the data machines/services, assuming they are somewhere else or replicated
• Or somehow push in data from a backup
• DNS is now a switch-over, rather than a ‘create’ operation

Maybe you practiced those too.

After the disaster

Assuming you still love your original hosting company, after they come back from their outage, you have source-control to lean on and a number of commits to rollback. It still requires a plan, and some sign-offs as execs will have frayed nerves, but you can go back methodically if you want.