Dr. Neil's Notes
Software > Development > AI in Delivery Systems
Metrics that matter
Introduction
Counting prompts, sessions, tokens used, or model calls can create the illusion of progress. Outcome-oriented teams measure changes in delivery performance, quality, and team effectiveness.
Track flow and quality together
Measure cycle time, lead time, and throughput alongside defects and rework. Faster without safer is fragile, safer without faster may not get further investment.
Use a small set of measures that show how work moves from idea to production. Compare similar work over time rather than treating a single result as proof of improvement.
Review the measures together. A shorter cycle time may come from smaller changes, or it may reflect skipped testing and review. Look for improvements that reduce delay without increasing defects, rollbacks, or rework.
Make the measures visible to the team and discuss them regularly. The goal is to find bottlenecks and improve the system, not to rank individuals.
Measure cognitive load
If AI introduces more review overhead than value, teams will feel slower even if some metrics improve. However, cognitive load extends beyond code reviews. Cognitive load includes understanding system behaviour, domain boundaries, how customers use the software today, and how that usage is likely to evolve in the future. Tracking time spent in manual verification, context switching, and domain comprehension may serve as key signals for measuring cognitive load.
However it is often difficult to quantify precisely and so the best approach is to talk to the developers, listen to their stories, understand how their work has changed. This cannot be achieved in a one-off conversation, it requires ongoing dialogue and observation.
If cognitive load is high, it may indicate that the system is becoming unnecessarily too complex. The best solutions simplify the system, reduce unnecessary context switching, and improve documentation and knowledge sharing.
Use metrics to decide, not decorate
Metrics should drive continuation, redesign, or retirement decisions for each new change. A metric that is measured but does not influence a decision is a poor metric, it consumes attention without helping the team learn or act.
Measurement also changes behaviour. People naturally optimise for the measures they believe they are being judged by, often in ways that satisfy the number rather than improve the outcome. For example, a focus on throughput can encourage smaller or lower-value changes, while a focus on shorter cycle time can lead to skipped testing, reduced collaboration, or work being split to make performance appear better. Treat metrics as signals rather than targets, review them in context, and pair them with quality and outcome measures. The purpose is to improve the system, not to win at the measurement game.
Part of the AI in Delivery Systems series.