statecraft.
Catalog/rustev

041-ranking-and-rubric-metrics

Ranking, constraint-match and rubric metrics; pluggable scorers (amends 004, 014 and 029)

Lifecycle approvedImplementation completeVerification unknownQualification unknown

An amendment of approved specs 004, 014 and 029 (R-16), for #76 child E1. Under a new `rustev.evaluator-config/3`, evaluation adds ranking metrics (precision, recall, nDCG, MRR and hit at k over graded gold relevance, with unjudged items excluded and counted, and per-position backfill rate), declarative constraint-match metrics declared in a new `rustev.task-adapter/2` and bound into evaluation identity, origin and evidence-quality rates for the constructs of #76 B1, B2, B5, B6, B7 and C2 (specs 036, 032, 037, 038, 035 and 039), and rubric metrics from pluggable scorers. Scorers are composed at compile time (a Rust trait, or a TypeScript object through the Node binding under R-38); their scores travel as `rustev.scores/1` documents bound to `rustev.score-views/1`. Model judges are ordinary plans with a `rubric` step whose retained bundles eval reproduces; their metrics are model-derived, named under `model/` and never pooled with deterministic ones, with judge agreement statistics. Gold records arrive in a new `rustev.dataset/2`. Every new metric defines its per-case measure, so spec 029's repeat statistics, intervals, slices and selection gates apply. Existing documents, identities, plans and CLI outputs keep their bytes. Approved (A-38); implementation is complete. Approval authorizes no live run, provider call or spend, and no quality claim (R-37).

Read the full pinned specification ↗

Relationships

Claimed units

  • extends
    {
  "nature": "amending",
  "spec": "004-evaluation-and-replay",
  "unit": {
    "kind": "directory",
    "path": "crates/rustev-eval/"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "006-cli-surface",
  "unit": {
    "kind": "directory",
    "path": "crates/rustev-cli/"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "027-node-and-wasm-bindings",
  "unit": {
    "kind": "directory",
    "path": "crates/rustev-surface/"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "027-node-and-wasm-bindings",
  "unit": {
    "kind": "directory",
    "path": "integrations/rustev-node/"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "031-public-api-and-release-family",
  "unit": {
    "kind": "directory",
    "path": "tools/rustev-release/"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "007-support-routing-package",
  "unit": {
    "kind": "directory",
    "path": "packages/rustev-pkg-support-routing/"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "008-reference-package-lodging",
  "unit": {
    "kind": "directory",
    "path": "packages/rustev-pkg-lodging/"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "001-boundaries-and-authority",
  "unit": {
    "kind": "file",
    "path": "Makefile"
  }
}
  • extends
    {
  "nature": "additive",
  "spec": "001-boundaries-and-authority",
  "unit": {
    "kind": "directory",
    "path": "scripts/acceptance/"
  }
}

Source headings

  1. 041: Ranking, constraint-match and rubric metrics; pluggable scorers (amends 004, 014 and 029)
  2. 1. Purpose
  3. 2. Territory
  4. 2.1 Binding principles of #76
  5. 3. Behavior
  6. 3.1 Documents and versions
  7. 3.2 Common rules
  8. 3.3 Gold records
  9. 3.4 Adapter measures: ranking, constraint and field declarations
  10. 3.5 Origin and evidence-quality metrics
  11. 3.6 Scorers
  12. 3.7 Scorers, views and scores
  13. 3.8 Model judges
  14. 3.9 Host scorers through the Node binding
  15. 3.10 CLI surface (adds to spec 006)
  16. 3.11 Per-case measures and spec 029
  17. 3.12 Gates and selection
  18. 3.13 Compatibility
  19. 4. Out of scope
  20. 5. Observable negative cases
  21. Acceptance
  22. Verification
  23. Implementation record
  24. Decisions at approval

Frozen anchors

No frozen anchors declared.