statecraft.
Catalog/rustev

029-evaluation-repeatability

Evaluation repeatability, baselines, diffs and selection gates (amends 004 and 006)

Lifecycle approvedImplementation completeVerification unknownQualification unknown

An amendment of approved specs 004 and 006 (R-16), for #76 child E2. Evaluation runs each configuration N times from a declared repeat source (deterministic replay, captured historical run sets, or live runs once spec 026 is implemented), reports per metric and per slice the mean, standard deviation, minimum, maximum, a pooled estimate and a case-clustered percentile bootstrap interval computed with a named, versioned, seeded generator, and derives declared noise bands. Adds named baselines (`rustev.baseline/1`), a reproducibility check, diffs in JSON and Markdown, multi-candidate comparison, held-out accounting against a host-kept ledger file, and a declarative selection gate (`rustev.selection-gate/1`) with stated CI exit codes. Every existing document, gate and CLI output keeps its bytes. Answers spec 023's open question 3. Approved (A-25); approval authorizes no live run, provider call or spend, no quality claim, and no change to `rustev.eval-report/1` or any plan identity.

Read the full pinned specification ↗

Relationships

Claimed units

  • extends
    {
  "nature": "amending",
  "spec": "004-evaluation-and-replay",
  "unit": {
    "kind": "directory",
    "path": "crates/rustev-eval/"
  }
}
  • extends
    {
  "nature": "amending",
  "spec": "006-cli-surface",
  "unit": {
    "kind": "directory",
    "path": "crates/rustev-cli/"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "001-boundaries-and-authority",
  "unit": {
    "kind": "file",
    "path": "Makefile"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "001-boundaries-and-authority",
  "unit": {
    "kind": "directory",
    "path": "scripts/acceptance/"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "027-node-and-wasm-bindings",
  "unit": {
    "kind": "directory",
    "path": "tools/rustev-conformance/"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "031-public-api-and-release-family",
  "unit": {
    "kind": "directory",
    "path": "tools/rustev-release/"
  }
}

Source headings

  1. 029: Evaluation repeatability, baselines, diffs and selection gates (amends 004 and 006)
  2. 1. Purpose
  3. 2. Territory
  4. 2.1 Binding principles of #76
  5. 3. Behavior
  6. 3.1 Repeat sources and runs
  7. 3.2 Per-case measures
  8. 3.3 Repeat configuration and statistics
  9. 3.4 Bootstrap intervals
  10. 3.5 Noise bands
  11. 3.6 Slices
  12. 3.7 Split roles and held-out discipline
  13. 3.8 Baselines
  14. 3.9 Reproducibility check
  15. 3.10 Diffs
  16. 3.11 Multi-candidate comparison
  17. 3.12 Selection gates
  18. 3.13 CLI surface (amends spec 006)
  19. 3.14 Exit codes
  20. 3.15 Compatibility
  21. 4. Out of scope
  22. 5. Observable negative cases
  23. Acceptance
  24. Verification
  25. Implementation record
  26. Decisions at approval

Frozen anchors

No frozen anchors declared.