Scoring

Definitions

For each scored rule r with weight w(r) ∈ {1, 2, 3}, applicable subjects are partitioned (after removing not-applicable ones) into direct passes, direct fails, inferred, and unassessed (not_evaluated or error). Each site-level rule contributes one site subject.

p(r)       = |D_f(r)| / |D(r)|                       for r with direct evidence
Readiness  = 100 · Σ w(r)·(1 − p(r)) / Σ w(r)         over rules with direct evidence
Coverage   = 100 · Σ w(r)·|D|/(|D|+|I|+|U|) / Σ w(r)  over applicable rules
Confidence = 100 · Σ w(r)·|D|/(|D|+|I|) / Σ w(r)      over rules with direct or inferred evidence

• Readiness is null with no direct evidence; coverage is 0 with no applicable rules; confidence is null with nothing evaluated.
• Unrounded throughout; display rounds half away from zero, once.
• Sub-checks. A fail on any evaluated sub-check fails the subject. All evaluated sub-checks passing while one applicable sub-check could not run is not_evaluated (incomplete_subchecks). A rule never passes on part of its own checks.
• Blocker cap. Readiness is capped at 40 when direct evidence shows robots.txt answering 5xx, a search crawler blocked site-wide against declared policy, or every route noindex. The uncapped value and the reason are kept.
• Display. Coverage ≥ 80% shows readiness; 50–80% shows it as provisional; below 50% (or no readiness) shows “insufficient evidence” and no number.

The score range

Every report also gives each rule one result: Clear, Problem, Can’t tell or Not applicable. From those, the CLI and the agent plugins calculate one score with one formula, shown as a range. The low end assumes every Can’t tell rule is a problem, and the high end assumes every one is clear. The CLI puts this in the report’s summary. Readiness below stays stricter: a rule never passes on part of its checks.

If bot protection or a firewall answers instead of the site, the report says it was blocked, sets blocked, and passes or fails nothing about the site.

Worked examples

These are executable fixtures in packages/geopilot/test/fixtures/scoring. The test suite recomputes every number below from them on every change, and this page reads the same files, so the prose cannot drift from the engine.

Readiness = 100 × (19 − 2 × 1/3) / 19 = 96.49 → 96; Coverage = 100 × 19 / 23 = 82.6% → 83%; Confidence = 100%

Uncapped = 100 × (27 − 3.5) / 27 = 87.04; capped at 40

Uncapped = 100 × (25 − 3 − 2 × 1/4) / 25 = 86; capped at 40; Coverage and Confidence 100%

No direct evidence: Readiness null, Coverage 0%, Confidence null

Readiness = 100 × (7 − 0.5) / 7 = 92.86 (hidden); Coverage = 100 × 7 / 23 = 30.4% → 30%: below 50%, so “insufficient evidence”; Confidence = 100 × 7 / (7 + 3) = 70%

Readiness = 100 × (3 × 2/3 + 2) / 5 = 80; Coverage = Confidence = 100 × (3 × 3/4 + 2) / 5 = 85%

Comparing runs

• A numeric readiness delta is shown only when the scoring version, ruleset major.minor and the set of direct (rule, subject) pairs are identical. Otherwise the comparison says “assessment scope changed” and lists what entered and left.
• A finding that moves from fail to not_evaluated or error is not resolved.
• Cross-site scores are not rankings. Even with an identical scope it is a technical readiness comparison, not AI performance.

Eleven properties are checked on 10,000 generated cases each: bounds, null-iff-no-direct-evidence, fail→pass monotonicity, unassessed evidence never improving coverage or readiness, inferred evidence never moving readiness, order and sample invariance, cap invariance, and the comparability rules above.