Requested, observed, effective: what an AI agent run proves

When a coding agent ends its turn with “all tests passed”, it has made a statement about a run. It has not produced evidence of one. The statement may well be true. It is also written by the same model that made the change, inside the same context, with every reason to describe its own work as finished. Treating that sentence as verification confuses a self-report with a record.

Kelruno records three different things about a task and keeps them apart:

Requested is what the task asked for: the model, provider, effort level, sandbox, or approval policy named in the request. A requested value is only a request. It is never promoted into a fact about what actually ran.
Observed is evidence recorded about the run at its actual strength. Observations come in grades, and a weaker observation is never rounded up to a stronger one.
Effective is reserved for the strongest grade: provider-native, invocation-bound proof of what actually served the run. When a proven effective value differs from the request, the effective value wins in audit and history.

The rest of this article is what follows from keeping those three words separate.

Grade	What it records	What it does not establish
Requested	What the task asked for	What served the run
Observed	Evidence at the strength its source supports	Anything stronger than its grade
Effective	Provider-native, invocation-bound proof of what ran	That the change is correct

Requested is not a fact

A task says: use this model at this effort level, inside this sandbox, under this approval policy. That sentence describes an intention. Between the intention and the invocation sit a provider, a tool, a routing layer, and a set of defaults, and any of them can end up serving something other than what was named.

The common shortcut is to write the request into the log as though it had happened. The log then says “ran with model X” because model X was asked for, not because anything showed that model X answered. Once a request is written in the same shape as a result, nobody reading the history later can tell the two apart.

Kelruno records the request as a request. It stays in the execution history in that shape, labeled as what the task asked for. That is useful in its own right: it shows what policy intended. It is not evidence of what ran, and it is not rewritten to look like evidence.

Observation has grades

“What actually ran” is not a single yes-or-no fact. It is a claim with a source, and sources differ in strength.

Some examples of the grades an observation can carry, in rising order of strength:

Accepted by the provider tool. The tool took the requested value without rejecting it. That shows the request was not refused. It does not show what served the invocation.
Reported by provider metadata. A value came back from the provider side after the call. That is stronger, because it did not originate in the request.
Proven by the trusted broker. The value was established for that specific invocation by the component Kelruno trusts to establish it. That is the strongest grade.

The rule that ties these together is short: a weaker observation is never rounded up to a stronger one. If the only evidence is that a tool accepted the request, the record says exactly that. It does not say “confirmed” because “confirmed” would read better. If a record is going to be quoted in an audit or a post-incident review, the grade it carries has to be the grade it earned.

Effective is deliberately narrow

It would be simpler to call every observation “effective” and move on. Kelruno does not, because the word has to mean one specific thing if it is going to carry weight later.

Effective is reserved for provider-native, invocation-bound proof: evidence that comes from the provider itself and is tied to the specific invocation, rather than inferred from the request or reconstructed from surrounding context. Only evidence at that grade is recorded as effective.

Two consequences follow.

The weaker observations do not disappear. A tool acceptance is still recorded as a tool acceptance. Provider metadata is still recorded as provider metadata. Every grade remains distinguishable in the execution history. Reserving “effective” for the strongest grade is a naming discipline, not a filter that discards the rest.

And when a proven effective value differs from the requested one, the effective value wins in audit and history. The request is still there, still labeled as the request. But the fact about what ran is the proven fact, and it is not overwritten by what was asked for.

When request and effective disagree

A requested/effective mismatch needs to be explained under policy. In Kelruno’s authoritative, broker-backed execution, an unexplained mismatch blocks the run.

That rule is fail-closed. The system does not pick the closest available substitute and continue. It stops, and the mismatch stays in the record.

The same principle governs routing. Where routing policy requires refusal, there is no silent fallback to a different provider, model, or effort level. Unsupported routes refuse rather than degrade. A run that quietly moved to something more available, then reported success, is the failure mode this design is built against.

Fail-closed is a scoped property. It applies to the execution paths where Kelruno’s policy applies. It is not a statement about every command a developer might run by hand outside the system.

Why the exact SHA matters

Evidence is only useful if you know what it is evidence about.

Every observation above, every exit code, and every gate outcome is a fact about one specific revision of the code. In Kelruno, review, remediation, and approval target an exact SHA, and evidence references name that SHA rather than a branch name or a conversation summary.

A branch name is a pointer that moves. “Reviewed the main branch” says nothing once the branch has advanced. A conversation summary is weaker still: it is the model’s memory of what it did, which is the self-report this article began by setting aside. An exact commit identity is neither of those. Bind the record to it, and someone reading the history later can check whether the evidence still describes the code in front of them.

Binding to an exact SHA also keeps a common gap visible. A review recorded against one commit says nothing about an amended revision. The amended revision is a new SHA, and any evidence about it has to be produced against that SHA.

Evidence is not correctness

This is the section that is easiest to skip and most important to keep.

Proving what ran does not prove that the code is correct. It does not prove the change is safe, free of bugs, well designed, or of high quality. A run can be fully accounted for, with an effective model recorded, every test exit code captured, and every digest bound to the right commit, and still have implemented the wrong thing, or the right thing badly.

Execution evidence answers a narrower question: what actually happened when this candidate was exercised. That question needs a reliable answer before a review is worth much, because a review of a run that did not happen as described is a review of fiction. But the answer to “what happened” is an input to judgment. It is not a substitute for judgment.

Work on coding-agent benchmarks makes the same point from a different direction. Wang, Pradel, and Liu studied patches counted as resolved on SWE-bench Verified and found that patches which pass the benchmark’s tests can still diverge from the behavior of the developers’ own fix, and that some are plainly incorrect. Passing the tests that exist is evidence about those tests. It is not proof about the change. Separately, Huang and colleagues found that large language models struggle to correct their own reasoning without external feedback, which is one reason a self-generated “looks good” carries so little weight on its own.

Kelruno uses execution evidence to support review. It does not use evidence to replace review.

Reviewer versus Verifier

Two roles are easy to conflate and are deliberately kept apart.

The Verifier runs the candidate’s own checks: its tests, its preflight gates. It reports exit metadata only. It holds no provider, repository, or infrastructure credentials, and it cannot approve or publish anything. The Verifier does not judge the change. It exercises the change and records what came back. Around it, the control plane records execution evidence: exit codes, stream digests, byte and line counts, and gate outcomes.

The independent Reviewer judges. It reviews an exact SHA it did not build, from a different provider/model family where policy requires independent review, and it has no authority to remediate that revision. A Reviewer that starts fixing the code stops being independent for that revision, so it does not fix. It reads, it checks, it reports.

The Builder is the third role: the agent that made the change and ran its own checks before handing it over. A Builder is never its own final reviewer.

Research on using language models as evaluators has documented a self-enhancement bias, a tendency for a model acting as judge to favor its own outputs, alongside position and verbosity biases. Zheng and colleagues describe these limitations in their study of model-as-judge evaluation. Using a different provider/model family for the final review, where policy requires it, is one structural response to that concern, as opposed to a prompt-level one.

Separating the roles gives the evidence chain a shape a person can reason about. The Builder produces a candidate. The Verifier produces facts about the candidate. The Reviewer produces a judgment about the exact revision, informed by those facts. Each role has a bounded job and bounded authority.

Human Gates

Merge, production deployment, and physical or network device changes are Human Gates: approvals only a person can give, recorded outside model authority. Automation gathers evidence and proceeds up to the gate. It does not pass through it.

That boundary is what the evidence is for. The person at the gate can see the requested values, the observed values at their real grades, the effective values where they were proven, the exit codes, and the exact SHA they all belong to. What that person should not have to rely on is a paragraph from the model explaining that everything went fine.

Current status

Kelruno is pre-release. It is not generally available. There is no public self-service signup and no published pricing. A gated, invite-only Founder Beta is planned.

The mechanisms described here are capabilities that exist today: graded requested, observed, and effective values in execution records; fail-closed, broker-backed execution; exact-SHA review and evidence references; separated Builder, Verifier, and Reviewer roles; and Human Gates for merge, production, and device changes. This article claims nothing beyond them.

References

Huang, J., Chen, X., Mishra, S., et al. (2023). Large Language Models Cannot Self-Correct Reasoning Yet. arXiv:2310.01798.
Wang, Y., Pradel, M., and Liu, Z. (2025). Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study. arXiv:2503.15223.
Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.