Paul Hammant's Blog: Build-time AI vs Run-time AI
I was at a ThoughtWorks “The Future of Software Engineering” event in Switzerland some weeks ago. What a braintrust. Lots of conversations about AI generally, agentic AI in particular and harness engineering.
Here’s a distinction I’ve been mulling for a while and keep reaching for in conversations about AI systems in the enterprise: build-time AI versus run-time AI. One is artifact-oriented, the other is execution-oriented.
| Mode | Primary artifacts | Versioned? | Typical outputs |
|---|---|---|---|
| Build-time AI | Source files, configs, prompts, AGENTS.md, tests, policies, IaC | Yes (Git) | Commits, PRs, CI/CD, deployments |
| Run-time AI | Live state, requests, conversations, workflows | Usually no | Database rows, tickets, CRM updates, emails, transactions, generated media |
Run-time AI isn’t only transactional, either. Generation of movies, songs, images, and slide-shows is run-time too. Those outputs are files, which makes them look artifact-like, but they’re products of execution, not engineering artifacts. Say a generated slide-show doesn’t specify the behavior of any system, and nobody’s putting it through code review as it might be going straight into GoogleDocs or Office 365. That’s the real test for which side of the line something sits on: build-time artifacts specify (later) behavior; run-time outputs are the consequences of wished-for behaviors.
Cross that with the conversational -> workflow -> agentic ladder that has become the common way of classifying AI systems, and you get a tidy 3×2:
| AI type | Build-time | Run-time |
|---|---|---|
| Conversational | Prompt engineering: templates, system prompts | A single chat session |
| Workflow | Workflow engineering: workflow definitions, tool configuration | Executing the predefined process |
| Agentic | Harness engineering: code, prompts, tools, memory policies, guardrails | The agent reasoning, planning, acting, updating systems |
Harness engineering is a build-time discipline
The interesting consequence: harness engineering is predominantly a build-time discipline, even though it exists entirely to influence run-time behavior.
The build-time harness looks like source code, because it is source code:
AGENTS.md
tools.yaml
memory_policy.py
planner.py
evals/
guardrails/
prompts/
Well, it is today. It gets committed to Git, reviewed in pull requests, and deployed via CI/CD - the full software delivery treatment.
Run-time is a different world entirely. Steps in a workflow:
- Customer asks for a refund
- Agent plans
- Looks up CRM
- Issues refund
- (Audit log written)
- (Database updated)
The last two are in parentheses because the agent isn’t doing them directly - the existing application or service (AI made or not) it invoked does those as a matter of course.
No Git commit occurs during that interaction. The agent changed the state of several systems of record, but it didn’t change itself.
An orthogonal axis, not a maturity stage
It’s tempting to present build-time vs run-time as another rung on the maturity ladder, after conversational, workflow, and agentic. I’d resist that. It’s an orthogonal axis. There are two independent questions to ask of any AI system:
- What kind of AI system is this? - Conversational → workflow → agentic.
- Where does the “engineering” happen? - Build-time artifacts ↔ run-time execution.
Every cell in the 3×2 above is a legitimate place to be, with the non-agentic ones included in that claim. A prompt-engineered conversational system with its templates under version control is not “less mature” than an agentic one. Indeed, it may be exactly the right amount of machinery for the job.
What a “harness” actually is
This framing also gives “harness” a satisfyingly software-engineering flavor. Edward Mangini’s recent Harness Engineering: The Devil is in Your Details defines a harness very broadly - essentially “everything except the model.”. He credits others (Trivedy & Birgitta Böckeler) for that. I think there’s a cleaner separation: the harness is the version-controlled, deployable specification of agent behavior. The live conversation, the database state, and the external side effects are not the harness - they are the consequences of the harness executing.
That gives harnesses a lifecycle human software engineers will recognize:
**author -> review -> commit -> deploy -> execute -> observe -> revise
Everything before “execute” is build-time. “Observe” is where run-time telemetry feeds back into the next build-time iteration. Which is to say: harness engineering is just engineering, and all the disciplines we already have for source code - code review, trunk-based development, continuous delivery, configuration as code - apply to it directly.
Well of course, build-time vs run-time isn’t common language in the AI field, so I’ll keep on trying to update my mental model.