ADR 0003: Results from outside, and the machines that produce them

Date: 2026-09-17 Status: accepted

Context

A check can only take part in scoring if a Python function for its key exists in a closed registry compiled into the application. So adding a scanner is a CRAne release, and a deployment that wants its own metadata-quality check has to wait for one.

Three open issues turn out to be the same mechanism read three ways:

• an outside scanner that reports results nobody asked it for;
• a small runner that executes work CRAne queued, where that work can run — a workstation, another network zone, an operating system the server does not run;
• an interpretive reading, where the work is an agent CLI reading a support statement and saying what it thinks.

All three are a program outside CRAne that declares what it can do, does it, and reports a result. What differs is who started it. The question this settles is what such a program may assert, what CRAne does with it, and what it is never allowed to move on its own.

The alternative considered and rejected is an executable plugin: an assessor uploads Python and CRAne runs it in process. It is the shortest path to “any check”, and it makes every other guarantee here unenforceable, because a plugin that can compute a verdict can also write one.

Decisions

1. A result is a statement about one subject, and only a statement closes a finding

An assessor is a registered thing with a name and a namespace, not a credential. Tokens are issued for it, and rotating or replacing a token does not make a second assessor: results and findings belong to the assessor, so a scanner whose token was replaced can still close what it raised last week.

An assessor submits, per result: the check key, the subject as a source system’s own identifier, an optional variant, one of the seven outcomes this product already has, an explanation, structured details, evidence references and when it was observed. How the result was produced is a property of the assessor rather than a field in the payload; anything the payload says about itself is carried as detail and decides nothing.

A finding belongs to the assessor that raised it: its identity is that assessor, the check key, the subject and the variant. It is closed when that same assessor reports pass about the same four. Nothing else closes it. Another assessor’s pass about the same check is its own statement about its own finding, and two assessors disagreeing stays visible rather than one clearing the other’s work.

pass is the only outcome that closes. not_applicable from an assessor is recorded and closes nothing, because a tool deciding a check does not apply here is a guess about applicability, while the same word from a person is a decision with a name against it — and collapsing the two would put an observation where a decision belongs. The outcomes that report a failure to look (source_inaccessible), an unmeasured result (insufficient_evidence) or one waiting on a person (review_needed) close nothing either.

Ordering is by arrival, not by the caller’s clock. The observation time an assessor supplies is recorded and is refused if it is in the future, because one bad clock would otherwise make every later result look older than a result that has not happened yet, and the finding it raised could never be closed again. A result that arrives after a newer one for the same identity is filed in history and moves nothing. A finding’s detection time is when CRAne received the result; the assessor’s observation time travels beside it.

Rationale: the obvious design is a sweep — a scan runs, and whatever it no longer reports is fixed. That reads an absence as evidence, which is the one inference this product refuses everywhere else: an empty field means nobody looked. A truncated scan, a crashed scanner and a clean portfolio all produce the same silence, and only one of them means the problem is gone. Requiring a positive statement makes a partial scan harmless by construction rather than by a flag the scanner sets about itself.

2. A re-raise never overwrites a person’s judgement

An assessor re-reporting a failure lands on the same finding by the identity above. If a person has closed that finding as accepted, not_applicable or duplicate, it stays closed and the result is recorded as still reported. Only a finding closed by that assessor’s own pass reopens when the failure comes back, with a new detection time.

Rationale: accepted means somebody looked and decided to carry the risk; not_applicable means they decided it never applied here. Both are decisions with a name against them, and the authority ladder puts a person above an observation. A scanner silently reopening them every ten minutes would erase the decision and the queue would never be answerable.

3. A run record measures coverage. It never closes anything

An assessor opens a run, declares the scope it intends to cover as a list of subject references — a source system and an identifier each, optionally narrowed to particular check keys — posts results against it, and closes it as complete, partial or failed. CRAne compares what a run declared against what it reported: a run that claims completeness having skipped part of its declared scope is recorded as partial, with a diagnostic naming what is missing. A run left open past its lifetime is abandoned by CRAne rather than left open forever.

The run answers “how much of the portfolio has anybody actually looked at, and when”. It is reporting, not enforcement. Given decision 1, nothing about closing depends on it, so a scanner that lies about its own completeness misreports coverage and cannot clear a finding it never looked at.

4. Assessors reach their own endpoints, not the import contract

Results arrive at endpoints of their own: open a run, post a batch of results, close the run, read the receipt, read the published schema. The parts of the staged-import design that earn their place are reused — an idempotency key, a receipt naming every row, an unresolved subject recorded rather than failing the batch, and work above a size threshold going to the job queue.

Three parts of the import contract are actively wrong here, which is why this is not one more dataset:

• the contract fingerprint couples every dataset’s schema to every other, so a results schema change would refuse unrelated imports;
• validate-then-apply is two calls, and a scanner posting every few minutes leaves a staged bundle behind every time it dies between them;
• an import answers a refused bundle with 201 and ok: false, which a retry loop reads as success. A scanner is a retry loop.

A schema-invalid batch is refused whole with a 422, because that is a bug in the scanner. A subject that resolves to nothing is accepted and named in the receipt, because that is a gap in the data and the two need different responses.

5. A token per assessor, scoped to a namespace and to source systems

An assessor authenticates with its own token, issued from the command line, shown once, stored as a hash, revocable in one row, and recorded as a decision when it is issued because a person decided to trust something. The token carries the namespace it may write checks under and the source systems whose subjects it may write about. Three predicates, and no role system: the namespace matches, every check key sits under it, and every resolved subject belongs to an allowed source system.

The source system named on a submitted subject is checked against the token before the subject is resolved, not after: a scope test that only covers subjects that resolved would let an assessor write about a system it may not touch by naming an identifier that does not exist yet.

A token carries an expiry as well as a revocation, and its last use is recorded. What an assessor is — an ordinary tool, or one that reads with a model — is a property of the token, not a field in the payload, because a payload field describes what the caller says it did.

On these endpoints attribution comes from the token rather than from a field the caller fills in, and the shared administrative token is refused for writing. It still reads: an operator has to be able to fetch a receipt and list runs without holding a scanner’s credential. ADR 0002 decision 1 said one shared token answers “may you reach this at all”; an assessor needs the narrower question answered, and leaving the shared token acceptable for writing here would mean nobody ever issues a scoped one.

6. Runners poll. CRAne never calls a runner

A runner asks for work it declares itself capable of, claims it under a lease, extends the lease while it works, and posts the result. The lease machinery, the expiry reaper and the attempt cap already exist for in-process workers and are reused. A job definition says whether it is executed externally, so in-process workers and runners never race for the same work. The capabilities a runner declares narrow what it may be given; its token bounds what it may be given at all, so a runner that overstates itself produces failed jobs attributed to it rather than access it should not have.

Rationale: the machine that runs this work is a workstation, behind NAT, on somebody’s desk, possibly holding collector credentials. A daemon that accepts “run this” is an execution endpoint on that machine, and CRAne would have to hold a credential to it. Polling needs no inbound port, no fixed address and no credential in that direction, and a queued job is picked up on the next poll, which is soon enough to feel immediate.

A runner returns the raw output of the job it was given, and nothing else. For work CRAne queued, the result is derived here from that output by the same parser an in-process worker uses; an outcome a runner states for itself is not read. So a compromised runner can misreport what a tool printed — which any tool can do on a machine somebody controls — and cannot assert a verdict, and a forged pass has nowhere to enter.

Which jobs a runner may receive is bounded by its token, which names the job definitions it may run and the source systems whose subjects it may be sent. The capabilities a runner declares narrow that further; they never widen it, so a runner that claims everything is still offered only what its token allows. A job payload carries identifiers and options, never a credential for another system.

Parsing the output of a tool CRAne queued stays here, which is where it already is: a runner executes and reports, and the parser per tool is the same in-process one an in-process worker uses today. That is not the registry this ADR escapes — an assessor with its own format does not queue work at all; it submits the normalized envelope of decision 1 and needs no parser here.

7. An external check may earn weight, through a policy, and may never be blocking

Results are findings and reporting signals first. A policy may then name an external check, declaring the assessor and the variant it reads and how stale a result may be, and it is evaluated by one generic evaluator rather than a function per key. A policy that marks an external check blocking is refused, and an assessor’s own numeric score is recorded and never scored — the outcome is what counts, and the weight is the policy’s.

A result older than the policy allows is insufficient_evidence, never a failure: an expired result means nobody has looked recently, and reading it as a failure would be the absence-as-evidence mistake decision 1 exists to avoid.

A person carrying a risk deliberately says so the way they already do, by granting an exception on that check for that subject, which the scorer applies before any evaluator runs and which no assessor can write. Closing an assessor’s finding as accepted or not applicable grants that exception in the same act, so the queue and the score cannot end up disagreeing about a risk somebody has already answered for. Without that, a repeated failure would keep docking weight after it was accepted, putting an observation above a decision.

Staleness is measured from arrival, the same clock the ordering rule uses.

Rationale: “a validator result never becomes a product compliance verdict on its own” holds because the verdict is the policy, not the result. Naming an external check is an edit to a versioned, hashed document that a person cut and that says which assessor it trusts. A compromised assessor can move its own checks’ weight on its own subjects, and is revoked in one row. Blocking is excluded because capping a band is the one effect that is not proportional to weight: it would let an outside tool fail a regulated product outright.

Interpretive assessors are narrower still. A result produced by a model may only be review_needed or source_inaccessible, refused on the way in, and a policy may give an interpretive check no weight at all. A model’s reading reaches a score only through a person deciding, which is the whole point of having one read at all.

Consequences

• An external assessor is useful before any of the runner work exists: a token, a run, results, and findings raised and closed by decision 1 stand alone.
• Coverage becomes a first-class question. “Nothing reported” is now visibly different from “reported clean”, and the run record is where that difference lives.
• Two assessors may implement the same check key in one namespace and disagree. Both results are kept and the disagreement is visible, the way two sources disagreeing about a claim already are.
• There is a new credential class to manage. Tokens expire, are revoked, and their last use is recorded; an unused assessor token is now a thing to notice.
• assessment_result, audit_check and job_definition.required_capabilities_json have been in the schema since the first migration, unwritten and unread. The capabilities column is what runners will use. The two tables are not the right shape: a result submitted by a scanner has no policy behind it, so this needs its own table, with a current row per logical result and an append-only history beside it. assessment_result and audit_check stay where they are, still unused, still describing the frozen-campaign record they were drawn for; taking them for something else would leave the schema describing a design nobody built.
• A namespace has one owner, and registering a second assessor into it is that owner’s call, recorded as a decision naming who agreed. From then on two assessors raise findings under the same keys — separately, by decision 1, and visibly.
• The interpretive path stops requiring a second worker with write access to the database, which is the cost docs/deployment.md documents today.