Dr. Neil's Notes
Software > Development > AI in Delivery Systems
Reliability guardrails
Introduction
This extends on the previous notes on human-in-the-loop architecture and outcome-first AI adoption. It is a practical guide to keeping AI-assisted delivery safe, reliable, and useful.
Note: This is not a guide to AI safety in general. It is a guide to keeping AI-assisted software delivery safe and reliable. The focus is on practical guardrails that can be implemented in a team, not on philosophical or existential questions about non-deterministic statistical probability machines, or what you probably call
AI.
AI-assisted delivery is a bit like giving a junior developer high-speed typing lessons and a very loud confidence level. It can write useful things quickly. It can also produce something that looks polished, is wrong in a way that is hard to spot, and arrives with the calm certainty of a person who has absolutely no idea what a production database is.
The point of guardrails is not to slow the whole machine down to a crawl. It is to keep failure small, visible, and recoverable. Reliability is not a moral virtue. It is a design choice. A team that is serious about AI delivery has to decide what the system is allowed to do, what it should check before doing it, and how quickly it can be turned off when the wheels come off.
If we do this badly, we get a familiar pattern; a system that seems clever, makes a few confident mistakes, and then quietly starts drifting away from reality. Guardrails are the thing that keeps that from becoming a production incident, a PR disaster, or a very expensive lesson.
Keep the blast radius small
The first rule is simple; do not let a new AI-enabled workflow touch the whole world on day one.
Start with a narrow problem, a small permission set, and a low-risk environment. Let it give advice before it acts. Let it draft before it ships. Let it suggest before it commits. This is not timidity. This is common sense with better packaging.
A good system begins with small bets. If the AI is helping write release notes, that is one thing. If it is generating migration scripts for production data, that is a different kind of sport entirely. The second one needs more checks, more evidence, and a person whose job description does not begin with "trust me, it seems fine."
A strong pattern is to separate "advisory" from "autonomous" modes. Advisory mode can be fast and broad. Autonomous mode should be narrow, bounded, explicit, and easy to stop. If the system cannot explain what it is doing and why, it probably should not be allowed to do it without a human in the loop.
This is not just about safety. It is also about learning. Small, contained experiments tell you whether the system is useful before it becomes dangerous. You do not need to discover the shape of the blast radius by accident. That is how people end up writing an apology email to the entire engineering organisation.
Use examples and tests like a sensible adult
Drift is the real villain in AI-assisted delivery. It does not usually arrive with fireworks. It sneaks in. A model starts acting slightly off. The prompt becomes a little more vague. A dependency updates. The evaluation set is stale. Then one day the team discovers the system is recommending the wrong pattern, missing a security control, or confidently inventing APIs that do not exist.
The answer is not mystical. It is boring, repetitive, and necessary; run regular evaluation tasks (or tests) against known cases.
This means maintaining a set of scenarios that matter:
- a good change request that should pass
- a common failure mode that must be caught
- a security-sensitive case that should be blocked
- an edge case that should trigger review or escalation
If you are working on an AI-assisted code review system, for example, you should test the AI code review on real examples of:
- a harmless refactor that should be approved quickly
- a bug fix that should be reviewed by an engineer
- an insecure pattern that should be flagged immediately
- a missing test that should lead to a stronger recommendation
These checks should be regular, not once-a-quarter "we'll do a proper review when we have time." Time is a bully. It never has a good plan. If quality slips, the system will usually tell you via its behaviour before it tells you via a dashboard. That often happens when a customer calls to say, "I think your system is broken."
A recurring eval set is a form of honesty. It gives the team a way to notice drift before it becomes a product-level incident.
Make the safety net real
Every AI-enabled change should be reversible. Not just in theory. In practice.
This means there should be clear rollback steps, staging environments, feature flags, and a simple way to disable the automated path. If the system can generate a pull request, it should also be possible to reset the branch, revert the change, or disable the automation without a heroic ceremony.
That sounds obvious. Yet many teams build AI systems with the enthusiasm of a toddler building a Lego tower and the operational discipline of a squirrel in a windstorm.
A simple example is an AI coding assistant that writes a migration or changes a deployment manifest. That is helpful until it commits a bad change to the main branch. The real question is not "can we trust the model?" It is "if it is wrong, can we undo it quickly enough that the team still has a lunch break?"
Good reliability is not only about preventing failure. It is also about recovering cleanly. The best systems do not just avoid bad outcomes, they fail in ways that are easy to contain.
Design for visible decisions
A guardrail is not a vague feeling that the team should be careful. It is a design choice that makes the system easier to interpret and easier to control.
That means three things:
- the system must know when it is uncertain
- the system must surface the reason for uncertainty
- the human or team must know what to do next
If the model is not sure, it needs to say so plainly. Not in a dreamy, abstract way. In a real way. Something like: "This change touches a production endpoint. The confidence is low because the surrounding context is incomplete. Human review required before deployment." Confidence scores are not enough. They are often misleading. A model can be very confident about a wrong answer. It can also be very uncertain about a right answer. The system that applied the model needs to validate the output.
This is not a sign of weakness. It is a sign that the system has a healthy relationship with reality.
The same goes for escalation. A workflow should tell the reviewer what to check, what evidence exists, and what the likely consequence is if the output is wrong. Otherwise you get the classic anti-pattern: a person receives a random blob of AI output, stares at it, and says, "This seems like it might the solution, but I am not quite sure why." That is not review. That is theatre.
Build the system to be honest, not merely impressive
The underlying pattern is simple. Reliable AI delivery needs a little humility.
A system should be clear about:
- what it knows
- what it does not know (when it knows it does not know something, which is rare)
- what it is about to do
- how it can be reversed
- who is accountable if it is wrong
That is the real guardrail. Not a wall of policy statements. Not a giant checklist no one reads. Not a dramatic "we are very careful here" poster in the engineering handbook. Real guardrails are simple, visible, and enforced in the flow of work.
A good AI delivery system behaves a bit like a sensible pair of shoes. It is there to help you move faster without falling into the road. It is not a magic wand. It is not a substitute for judgment. It is a tool that works best when the team knows what it is for, what it is not for, and how to stop it before it gets creative.
That is what reliability looks like in practice: small scope, the right checks, reversible change, visible uncertainty, and clear ownership. Not a perfect system. A high quality and safe one.
Part of the AI in Delivery Systems series.