Paul Hammant's Blog: Better practices for audits

Software development teams quest for continuous deployment these days. Some of those want to get there and to also pass audits they may be subject to. Audits can be from a firm that the company appoints (to the agreement of their clients), or by a state or federal overlord.

If the CD-questing team is using Git, has achieved continuous review, with CI kicking in for every commit that lands in trunk/master (or small batch of commits), and has a policy of all commits going green or they get automatically rolled back, then they have the perfect setup for audit-ready software.

Merkle Trees

All this rests on Git being a perfect history tracking Merkle tree for directories of source files over time. Git makes SHA1 hashes of the versions of source files and directories landing in a branch (‘master’ is the trunk equivalent in Git for Trunk-Based Development teams). If all commits that are in master are good (if not rolled back), and that means that all the tests of “the build” passed, then you have something that is hypothetically reproducible in an audit. Specifically, in an audit cycle, the auditor says “show me the test evidence for commit 25b23067de4004db8b4fcb2ad23f0dac16105b89” and an experienced developer at a dev workstation rolls back the source tree to that commit, and runs “the build” to the same outcome. Most likely the auditor has some development skills too, so they can smile their way past small build snaufus like “I need to reinstall JDK 8, it won’t take long”.

Git’s Merkle-tree implementation is so perfect (provided force-commits are turned off) that even the commit messages are subject to same tamper-evident design.

Works with any branching model

This model for audits works for Trunk-Based Development with:

  • Branch for Release for slower cadences
  • Release from tag (on trunk/master) for faster cadences
  • Release from commit (on trunk/master) for faster cadences

It also works for other branching models provided that the commit passing/failing CI is noted somewhere non-repudiable and the agreed audit process details the process of release and the policy in source-control for what is releasable.

The Merkle tree aspect to this proposition is so mathematically solid that it would cost upwards of a million dollars to adequately manipulate the history in Git to the level where the SHA1 has reported to auditors still exists. Google made a proof of concept collision for SHA1 hashing. The technique is more than what you’d need to insert a bogus file in the history to correct a parent SHA1 that in turn would allow the SHA1 of the build that went live to remain the same after the tampering, but the effort to back calculate what that should be could well be higher (and could masquerade as a JPEG for example) Git should perhaps change to SHA256.

Continuous Review Again

Continuous Review was pioneered by Google internally in their ‘Mondrian’ system, delivered for the masses by GitHub from its 2008 launch. If you’re using GitHub (the public portal or the enterprise app that you install on-premises), it stores the code reviews in a Postgres database that you don’t have access to. Well, you do for the on-prem install (in an appliance) but you have to reverse engineer the schema in order to tamper with the record. It would be better if the reviews themselves were in source control.

I’ve made the case for code reviews in source-control to the GitHub people, but couldn’t convince them. You’d get to take advantage of the same Merkle tree concepts to be able to slide smoothly into the audit cycle with a single SHA1 as the representation of the last code review commentary that made it to that release. If in the same repo/branch then the SHA1 would be for the source code itself and the reviews the same. Yes, as single SHA1 again.

The way it used to be

It used to be that test evidence was stored somewhere non-repudiable. If your tests were always green (passing) then you were always redundantly storing evidence (build logs, test reports, videos). If your tests were not always green then I maintain that you should not be going live with that build. I’m often asked what percentage of test failing is acceptable and the answer is always zero. In ThoughtWorks I never got to see a situation where the team was not collectively pushing for 100% passing, so it a bit of a shock after ThoughtWorks to encounter teams that thought that 80% was good (provided an expert interpreted them daily).

The old way was a lot of bullshit, and vulnerable to dozens of ways to fake evidence/record. Companies used to rest on the fact that the software that was used to attest the record towards an audit was approved somehow. That it has passed some scrutiny and was able to be relied on in the audit cycle without the auditor scoffing at it. In reality, evidence was fakeable if needs be. The costs for which could be $1,000 to $20,000 depending on what was being tampered with after the event.

Git’s internal implementation

These two blog entries details some of the Merkle-tree implementation of Git.

The first doesn’t mention the Merkle-tree aspect, but the latter does. Neither mentions Blockchain either (thank Turing). I say that as there is way too much presenting of Merkle-tree solutions as Blockchain in the current age.

Published

May 30th, 2018
Reads:

Categories