Paul Hammant's Blog: Environment Hell

This is out of date by many years, and companies that have this environment setup don’t do it this way anymore. Or they are losing market stare. Or they have lost all market share, so to speak. It was typical of a company that scaled its production capacity before the horizontal-scaling and cloud era kicked in. This would be after 1995, and should have ended before 2015, really. The rigid rules and broken promises around environments suffocated their delivery efforts. Many companies were like this, with big program management office teams and plenty of coordination and cajoling around planned release dates. Release Cadences couldn’t get above quarterly most likely.

Produced by OmniGraffle 7.12.1 2019-11-20 20:22:04 +0000 Canvas 1 Layer 1 Load Balancer (F5) Firewall Load Balancer (F5) Firewall Load Balancer (F5) Firewall Shared DB and services (not shown) C3 C2 C1 C0 Regular "in-dev" usage, sometimes “integration” Formal Pre- Prod Identical to Prod (one day after go-live) Gated “Certification” Environments Tier 1 Tier 2 Tier 3 Formal QA

Shapes are arbitrary representation of services. More that one of the same shape indicates more than one of the same thing in the same tier. F5 (load balancer) had knowledge/config about how many of each underneath in order work as needed. Where’s there one of a thing in an env, it is functional correctness that can be verified (ideally) in that env, and non-functional correctness has testing deferred towards envs that are more like production.

The Firewall and F5 is shared for all envs, and all tiers. Rules/config were very carefully reviewed. Often there may be no side firewalls - Something from (say) C3 may accidentally be configured to use services from (say) C2.

Observations:

  1. Developers & Testers where beholden to these envs - they had no ability to deploy/test on their dev workstations.
  2. Developers could deploy into C3 themselves - for whatever reason they needed to.
  3. C3 was pretty much unusable because of that and old hands (and special teams) may conspire to skip deployment to C3 and aim at C2 fir their first deployments/tests instead.
  4. Shared DB and shared down-stack services are an obvious disaster.
  5. Related: nothing from the world of modern DB migrations has any meaning here.
  6. If there’s a Build Server it is not doing anything beyond compile and pure unit tests. It may be batched, or per commit (as modern CI dictates), but so many testing stages are still manual or wholly decoupled from this setup, it is near to useless.

After interview, you’d lean that it would take three months to build another pre-prod sized environment and that it would cost $500K or more. And that immediately after that to make yet one more would yield no significant saving in time.

Produced by OmniGraffle 7.12.1 2019-11-20 20:22:04 +0000 Canvas 2 Layer 1 Load Balancer (F5) Firewall Load Balancer (F5) Firewall Load Balancer (F5) Firewall Shared Live DB and services (not shown) Staging 1 Pilot Prod Staging 2 (post live) 20% of live traffic, as pilot Production Will go live: non public testing for real data/services pre-prod Identical to Prod (for patches) Production Environments Tier 1 Tier 2 Tier 3

Observations:

  1. Outside of a planned or unplanned release “Pilot” may have 0% of the trafic. It’s duty was for the planned production pushes, even if it were provisioned 365 days a year.
  2. “Staging 1” may be frozen ahead of a use in a planned go-live progression (with “pilot” being next), while simultaneously “Staging 2” is in use for a bug-fix that couldn’t wait to go with the currently planned release (or may have to get re-applied/merged into the planned release too).
  3. Again this ends up reinforcing non CD release cadences and non CI ways of working.

So many clients over the years have see these pictures, that I figured I’d just make a blog entry.

Published

November 20th, 2019
Reads:

Categories