An API changes. Something has to decide what that means: whether to test it, what to test, how hard. For years the answer was a person: an engineer who knew the system, read the diff, and used judgement. That person does not scale to a world where code changes by the hour, written by agents, across an estate nobody holds in their head.
So you build a sentinel. An autonomous decision engine that watches for change, decides what to test, and issues the commands: no human in the loop for the routine case. The loop is simple to state: evaluate the change, decide the tests, command the run.
The hard part is not building that loop. The hard part is making sure it does not slowly turn into a machine that cries wolf.
An autonomous watcher drifts by default
A system that makes decisions without feedback does not stay accurate. It drifts. It develops opinions it never checks against reality.
Say the sentinel decides a certain kind of change is risky and fires a heavy test battery every time. If half of those runs come back finding nothing, it is wrong half the time, but nothing tells it so. It keeps firing. The team learns that its alerts are mostly noise, and they do what humans always do with a noisy alarm: they stop listening. At which point the one real finding arrives in a stream of false ones and gets ignored with the rest.
A watcher that never learns from what it found is worse than no watcher, because it spends trust it has not earned and then spends the trust of the real signal too.
Grade yourself against what the scan found
The fix is a feedback loop, and it has to close on reality, not on the decision. After a run, the sentinel learns what the scan actually found. That outcome feeds back into how it decides next time.
If a class of decisions keeps producing runs that find nothing, the engine should get less eager there: tune its sensitivity down, stop spending test budget chasing a shadow. If a class keeps surfacing real defects, it should lean in. The reference point is never the engine's own prior confidence. It is the evidence the scan produced. You calibrate against outcomes, or you calibrate against your own imagination, and only one of those gets more accurate over time.
This is also where change intelligence earns its place. The sentinel is not reacting to a vague sense that something moved. It is reacting to specific, detected change (a breaking schema change, a deprecated operation, behavioural drift away from a baseline) and then checking whether its reaction was warranted by what the tests found. The input is precise and the feedback is precise, so the calibration has something real to bite on.
Watch the watcher
Here is the subtle part. A self-tuning system can tune itself into a corner. So you need signals about the signals: a layer that monitors the health of the monitoring.
Track the false-positive rate over time. If it climbs past a threshold, that is not a normal finding; it is the engine telling you its own judgement has degraded and needs attention. Track staleness: feedback that has gone quiet means the engine is adjusting on old information, and old information is a quiet way to be wrong. Add cooldowns, so the system cannot thrash its own sensitivity up and down on every new data point.
Signals about signals sound like over-engineering until you have watched an autonomous system confidently optimise toward the wrong target. The meta-layer is what catches the engine before it drags everyone else down with it.
Young systems should not act like old ones
The most dangerous moment in the life of a self-calibrating engine is the beginning, when it has almost no feedback.
With a handful of outcomes to learn from, every adjustment is an overreaction. One false positive in three runs is not a 33% false-positive rate; it is three data points, which is nothing. A system that calibrates aggressively on thin data will swing wildly, and it will do it with full confidence, because it does not know how little it knows.
So calibration has to be gated by maturity. Below a threshold of accumulated feedback, the engine does not trust its own corrections; it holds a sensible default and refuses to over-fit to noise. Only once it has seen enough does it start adjusting in earnest. A cold-start system and a mature one should behave differently, because they know different amounts, and pretending otherwise is how you ship a brand-new engine that is sure about everything and right about nothing.
The courage to say "I'm not sure"
All of this rests on one discipline that is easy to state and hard to hold: low confidence must stay visible. It must never be rounded up to a pass.
A run the engine is not sure about is a warning, not a clean result. A run that could not cover what it meant to cover reports the gap rather than the green. This is the same rule that governs every verdict in Dino (an incomplete run returns no score rather than a misleading one), and it is exactly the rule a calibrating sentinel is tempted to break. The whole point of the engine is to reduce noise, and the laziest way to reduce noise is to stop reporting the uncertain cases. That is not calibration. That is going quiet, and going quiet is how a watcher fails in the dark.
Admitting low confidence is what keeps the feedback honest. You cannot learn from a mistake you relabelled as a success.
Learning is the feature
It is tempting to measure an autonomous testing engine by how clever its initial decisions are. That is the wrong metric. A clever engine that never checks itself is a confident liar waiting to happen.
The engine worth building is the one that is wrong, notices, and is less wrong next time: that calibrates against what the scans actually found, watches its own judgement for decay, stays humble while it is young, and keeps uncertainty on the record instead of hiding it.
A sentinel that cannot learn from its mistakes is just an alarm. The learning is what makes it a sentinel.