Skip to content
Inkstand

What a result establishes

A rule that produced no finding can be in any of five states, and only one of them is a pass. This is what each one means, how the counts add up, and why they are never collapsed into a percentage.

Five states, and one of them is a pass

Every rule in force at the moment of a run ends in exactly one of these. They are exhaustive and they do not overlap, which is what lets a release record account for every rule the account set rather than for the ones that happened to fire.

verifieda pass is a fact
Code checked the rule and the check IS the rule. A match is a fact about the words, and so is a clean sweep. A release can be blocked on one of these without anybody’s opinion.
sweptfound nothing, proves nothing
Code ran a real sweep and matched nothing — but the pattern is a proxy for a wider statement, so finding nothing establishes that the pattern did not match and not that the rule is satisfied. Reported as partially checked, never as clear.
reviewedan opinion
A model read it. Useful, and it can be wrong. Never promoted to verified, whatever it found or did not find.
steerednothing looked
In force, queued for judgement, and no code and no model ever inspected it. The rule went into a prompt and the prompt did not run. This is the state most tools in this category have no word for.
out of scopedeclared up front
In force, and the run said before it started that it would not be looking. A Word add-in and a visual rule; a key an owner narrowed; a rule scoped to a content role the request did not carry.

Four of the five are compatible with an empty findings list. A report that shows the findings and not the states has shown you a quarter of the answer, confidently. That is the failure this model exists to make impossible to ship by accident — the argument is at swept is not clear.

How the counts add up

Four lists come back from a run, and they are disjoint: verified, reviewed, steered and out of scope together account for every rule in force. A fifth — partiallyCheckedRuleIds — is a subset of steered and not a fifth bucket, so adding it to the other four double-counts.

The arithmetic
rules in force              14

verified                    6   code decided it
reviewed                    2   a model read it
steered                     5   nothing looked
out of scope                1   declared, not looked at
                           ──
                           14

of the 5 steered:
  partially checked         3   swept for something, found nothing
  never touched at all      2

The subset is deliberate and it errs in the one direction a compliance record must. A partial sweep promoted out of steered would make every “needs judgement” counter under-report, which is the reassuring direction. The cost is that every reader has to subtract, so the subtraction happens once, in brand/tiers.ts, and both the record and the screens ask that module rather than doing it again.

The five are never collapsed into a percentage. “86% compliant” is a number with no referent — it averages a fact about words with an opinion from a model with a rule nothing read, and the only thing it reliably communicates is confidence.

Detecting is not clearing

Whether code can find a violation and whether finding none clears the rule are two questions, and for 3 of the 9 rule kinds they have different answers.

Finds it and clears it4 kinds
The pattern is the rule. A banned phrase either appears or it does not; a required disclaimer is present or absent; a reading level is above a ceiling or below it; a closed period contains a date or does not. Findings are verified and a clean result is a real result.
Finds it, cannot clear it3 kinds
The pattern is a proxy. Four regexes over personal data establish that four patterns did not match, and not that no personal data is present — a named hospital, a rare condition and a treatment date identify somebody with no email address in sight. Findings are still verified; the rule is not.
No pattern reaches it2 kinds
“Must be substantiated” is not a string, and a free-prose house convention carries no field naming what to check for. Working out what to look for is itself the reviewer’s job, before any checking begins.

So a finding’s tier and a rule’s coverage are separate facts and the audit has to be able to state both at once. Flattening them would force a choice between two lies: calling a rule verified when nothing established compliance, or running no code at all on a rule where a sweep would have caught something real. The kind-by-kind detail is in the rule catalogue.

A rule nothing looked at, declared up front

Out of scope is the newest of the five and the only one that is decided before the run rather than by it. Three things put a rule there.

surface
The caller cannot see the thing the rule is about. A plain-text call has no colours in it, so a rule about a palette cannot be evaluated whatever anybody would like. Permanent, and nobody’s mistake.
configured
Somebody narrowed what this key or this request checks. Reversible, and a governance act — changing what a key checks is recorded on the account’s register.
no content
The rule is scoped to a content role and the run carried nothing of that role. A caption rule over a deck with no caption has not passed; it was never applied.

The third one used to be counted as verified — the strongest claim this model can make — and it flowed onto release records as “N rules verified in code”. A positive false statement in a compliance record is worse than any silence, which is why it is now a state of its own.

The reason travels beside the count rather than splitting it, because the reason changes who fixes it and changes nothing about the arithmetic. Over the API this arrives as scope.outOfScope, one entry per class with its reason, and outOfScopeRuleIds for the rules themselves. See the review API.