The Harness Passed. The Claim Did Not: Why Independent Evidence Matters

A synthetic qualification harness passed its tests. Its snapshot was stable. Its validator rejected malformed inputs. Its result record was well-typed. The local evidence looked impressively complete.

Then final review asked a narrower question: where did the claimed runtime observation come from?

The answer changed the verdict. The field labeled as an observation was derived from the same scripted input that expressed the intended result. The harness had proved that it could preserve and compare its own data. It had not proved that a runtime independently attempted or observed the claimed behavior.

A test can be internally consistent and still lack an independent measurement.

The Three-Layer Claim

Many agent-runtime claims quietly combine three different facts:

A policy unit test may establish intended. A mocked adapter may establish the shape of attempted. Neither automatically establishes observed.

How a Harness Accidentally Observes Itself

Consider this simplified anti-pattern:

const expected = testCase.requestedRoute;
const decision = chooseRoute(testCase);

return {
  intended: expected,
  attempted: decision.route,
  observed: testCase.requestedRoute
};

The result may satisfy intended === attempted === observed. But the equality is circular. The “observed” value came from the fixture, not from a runtime-owned observation boundary.

A stronger shape keeps the sources separate:

const intended = policyDecision(testCase);
const attempt = await dispatchBoundary.invoke(intended);
const observed = await observationBoundary.read(attempt.id);

assertEqual(intended.route, attempt.route);
assertEqual(attempt.route, observed.route);

This pseudocode is a design sketch, not a universal implementation recipe. The important property is source independence: mutating the fixture alone must not be able to manufacture the runtime observation.

An attempt ID is a correlation handle, not proof of independence. If the observation store merely copies the dispatch envelope, the second interface still reads the same claim-producing data path. A stronger measurement is produced by the downstream component that performed—or failed to perform—the behavior, then correlated to the specific attempt with freshness and request-identity checks. A mock can preserve that producer separation, but a mock that returns the fixture value proves only the simulated contract.

Use an Evidence-Independence Matrix

Before trusting a polished qualification packet, map each claim to its producer and ask what would falsify it.

The last row matters. A test suite that proves only the enabled path may miss restoration defects. A negative control shows whether the system can distinguish “the feature selected this path” from “the fixture always says this path.”

Mismatch injection and disabled-mode controls are necessary falsification checks, not sufficient proof of independent provenance. Two interfaces over one underlying envelope can pass both. The evidence still needs a trustworthy producer, correct attempt correlation, and a freshness boundary that rejects stale, replayed, or cross-request readback.

Validate the Measurement, Not Just the Result

Agent workflows often spend most of their effort validating artifacts:

• snapshot hashes;
• file-set and mode checks;
• schema validation;
• symlink and path protections;
• negative fixtures; and
• non-interference tests.

Those controls can be valuable. They establish integrity of the qualification machinery. They do not establish that the machinery observes the system under test.

Add four measurement questions to the review:

1. Producer: which component produced this field?
2. Independence: can the test input or dispatch envelope directly determine it without the claimed downstream behavior occurring?
3. Correlation and freshness: does the readback belong to this exact attempt, or could it be stale, replayed, or attached to another request?
4. Mismatch: has the harness demonstrated that it can detect intended, attempted, and observed values diverging?

If the answer to the mismatch question is no, a green equality check may be a tautology.

Let Final Review Produce a Real Stop

The clean response to an unproven central claim is not always another harness revision. Sometimes the workflow already promised:

one final independent review
at most one bounded execution if accepted
any decision-changing review failure => terminal stop

That rule changes incentives. The reviewer can say no without automatically creating a successor phase. The builder cannot keep adding proof machinery until the packet becomes too expensive to reject. The operator receives an honest outcome: useful offline evidence exists, but the runtime claim did not cross its evidence boundary.

A terminal stop record should contain:

• the exact claim that remained unproven;
• the decision-changing findings and their evidence;
• what the completed harness still proves;
• what was explicitly not executed or changed;
• the authority boundary that prevents automatic continuation; and
• the separate decisions available later: accept the narrower evidence, pause, or authorize a clean-sheet design.

This is not failure theater. It preserves both the value of the completed work and the truth about its limits.

When To Repair, Re-Anchor, or Stop

When Not To Demand Runtime Evidence

Not every useful claim needs a live or end-to-end runtime observation. Keep the proof proportional:

• a pure policy function can be proven with deterministic unit tests;
• a parser contract may need fixtures, not a live service;
• a schema migration may be adequately tested against an isolated copy;
• an exploratory prototype may explicitly claim only structural feasibility; and
• a high-risk live probe may be unjustified when the narrower offline evidence is enough for the current decision.

The mistake is not using synthetic evidence. A synthetic harness can preserve independent seams and detect a downstream mismatch. The mistake is attaching a stronger runtime claim to evidence that never crossed the relevant behavior boundary or came back through an independent producer.

Conclusion

A disciplined harness can prove many important things: deterministic policy behavior, artifact integrity, negative-input handling, non-interference, and a bounded execution contract.

It cannot prove an independent runtime observation by copying the intended value into an observation field.

Separate intended, attempted, and observed evidence. Name the producer of each field. Prove attempt correlation and freshness. Demonstrate that the measurement catches downstream mismatches while intent and dispatch metadata remain intact. And when final review falsifies the central claim, let the stop rule stop the workflow.

A green harness is evidence. It is not permission to claim more than the harness observed.