Rust application chassis
rahi
A chassis for governed cells, with storage, identity and delivery seams.
Observed 2026-10-03 · Producer spec-spine 0.28.0
73e937cce7f9d5c25db0939801456b8c94b2359e ↗36 specifications. Verification and deployment qualification are unknown in this export.
Bootstrap spec system for rahi (specify first, build by spec)
Foundational contract for the rahi corpus. Authored truth lives only in markdown with YAML frontmatter; machine-consumable truth about the corpus is compiler-emitted JSON read only through spec-spine; every artifact is a deterministic function of (config, file contents); and a typed authority graph governs who owns what. rahi is specified in full before a line of it is built: the corpus is the design, spec numbers are the build order, and an orchestrator drives one spec per fresh session through build, ship, shepherd, and verify. This spec also freezes the seven chassis invariants (constitution VI through XI and XIV) that no later spec may amend.
Agentic engineering harness: session protocol, skills, agents, hooks, gate
The governed-development loop every human and every driven session runs inside: the cross-agent New Sessions protocol and the Working the backlog protocol in AGENTS.md, the Claude Code skills (the eight that sequence the backlog protocol, prime, setup, next, build, verify, spec, ship, shepherd, and the two it calls, commit and code-review), the four pipeline agents, the standing and path-scoped rules (including the chassis invariants checklist), the hooks that report derived-artifact freshness and block an ungated PR, the Makefile that is the one source of truth for what CI validates, and the CI workflow that re-runs the same gate. The harness is what makes the corpus buildable by claude-observatory.
Chassis thesis: seven responsibilities, one deployment unit, a library not a template
The architectural record every ordinary spec operates inside. rahi is a set of Rust library crates and one binary composer that give an application exactly seven things: identity, state, ledger, kernel, edge, operations, and a dev substrate. It ships as one container with one volume and one origin, with rauthy co-deployed and never shared. It is a library that apps compose, not a template that apps are stamped from. This spec fixes the responsibilities, the crate topology, the dependency direction, and the three-wave build order. It owns no code.
Cargo workspace and the core types: Error, Principal, Revision, FenceToken, Config
The first build: a virtual Cargo workspace with the toolchain pinned, unsafe denied, one shared dependency table, and a supply-chain policy; plus rahi-types, the plain-data substrate every other crate depends on and that depends on nothing in the workspace. It fixes the Error enum with the four exit codes, the Principal as the IdP's subject plus claims, the Revision and FenceToken newtypes the store's invariants are written against, the Config tree derived from one public URL, and the schema version constants.
The store: hiqlite in-process with txn, two read calls, migrations, and backup
The application store: hiqlite linked as a library, opened from the Config with its own data directory and ports (never rauthy's), exposing exactly the surface the interface contract decided: execute, txn as the write path for anything with an invariant, query on the local replica and query_consistent through the leader as two named calls, migrations as versioned DDL run by a verb and never at boot, and the leader-only encrypted backup surface. Carries enrahitu://032 §3.1, §3.3, §3.6, §3.8.
Store coordination: fenced leases, local notify, the outbox, and the revision watermark
The coordination plane over the cache Raft group: distributed leases with the Raft log id as a mandatory fencing token and an explicit release, key-only local notify whose envelope is the routing key, the outbox row that commits in the same txn as the resource it describes, and the monotonic revision watermark every consumer polls so that notify is only ever a latency hint. Carries enrahitu://032 §3.2, §3.4, §3.5, §3.10.
The decision chain: hash-linked records, CAS append, verification at boot
The governance ledger: a linear chain of signed, hash-linked Decision records stored in the app's hiqlite, appended by a compare-and-swap that a unique index on the parent hash enforces inside one txn, retried after a bounded wait on a miss and then a typed integrity error, and verified in full at every boot with an integrity failure being process-fatal. Records use the attest-ledger envelope and canonical-keysort-json bytes so an independent verifier can check the chain without this code. Carries enrahitu://024 and enrahitu://032 §3.7.
Ledger sealing: a resident hot window, immutable segments archived to object storage
hiqlite replicates the whole database to every node, so an unbounded audit history cannot live in it; the head and a hot window do. When the window exceeds its bound the oldest contiguous run is sealed into an immutable segment (first id, last id, count, segment hash, previous segment hash), written to an object store behind a trait with an S3 implementation, and deleted from the hot table. Segments link to their predecessor and the head links to the last segment, so the chain verifies end to end without archived history resident. Archive is never conflated with backup. Carries enrahitu://032 §3.7 and §3.9.
The kernel: a declared capability ceiling, verified at build, enforced at runtime, denials ledgered
The governance seam. An app declares a manifest (TOML, embedded at compile time): identity, services, resources by name, a capability catalog, per service grants, the gate roster, ledger and observability policy. The kernel canonicalizes and hashes it, verifies at build that every governed facade the app links is covered by a grant, adjudicates every store, lock, notify, secret, and egress operation at runtime through the action-gate primitive, denies anything not granted, and appends a Decision for every denial without the request path awaiting the ledger. Carries enrahitu://020 and enrahitu://021 with the parser-based extractor replaced by a declared manifest.
Binary values and the extension policy: BLOBs are first class, loadable extensions are refused
Applications above the chassis store vectors, thumbnails, and signatures, so the store must round-trip binary values with a declared ceiling and a paged read that never materializes a table. It must also answer, once and in writing, whether a loadable SQLite extension may be used: it may not. hiqlite replicates statements rather than pages, so a node whose engine differs diverges silently, and a virtual table's behavior under snapshot, backup, and restore is unproven. The escape hatch is named here so a later spec can open it with evidence rather than by assumption.
The edge: axum router, middleware chain, probes, static slot, error mapping
The one public listener of a cell: an axum Router builder that an app mounts its services on, a fixed middleware order (observation outermost, then security headers, CSRF double-submit, and a per-node rate limiter on the cache group), liveness and readiness probes that answer different questions, a static file slot for the app's SPA, and one mapping from rahi_types::Error to HTTP status. Everything an app serves goes through this router; there is no second listener.
rauthy behind a same-origin proxy: the raw /auth/* route, discovery, JWKS, client bootstrap
One public origin for app and IdP: rauthy binds loopback, the app reverse-proxies the /auth/* subtree to it raw and unfiltered with bodies streamed, the issuer and every OIDC endpoint live under the app's URL, the driver trusts rauthy's discovery document fetched through the loopback base and caches its JWKS with rotation, and the app's OIDC client is bootstrapped on first boot with authorization-code plus refresh, S256 PKCE, RS256, and redirect URIs derived from the public URL. rauthy is a release binary in the same container and is never forked. Carries enrahitu://005.
Sessions and the principal: the signed envelope, renewal as a round-trip, the IdP subject as the only id
The app keeps the shell and gives up the authority. Login is the authorization-code flow with PKCE through the proxy; the session cookie is a signed envelope holding rauthy's refresh token and the pinned subject, httpOnly and same-origin, that grants nothing on presentation; the access assertion is short-lived and minted from the IdP's claims; renewal forwards the refresh token to rauthy and re-reads roles and email_verified from userinfo every time, so a role removed at the IdP takes effect within one access lifetime and a refused grant ends the session. No local account row is ever written. Carries enrahitu://004.
Observability: /metrics always on, an in-process OTel tracer with a ring buffer, decisions on spans
The substrate contract every cell exposes: Prometheus text on /metrics from an in-process registry (process metrics, HTTP counters and duration histograms labeled by route pattern and status class, store operation counters, kernel decisions and ledger failures), and an OTel tracer with a span per request and per store operation that exports OTLP only when an endpoint is configured and always retains a bounded ring buffer of recent traces queryable in-process. Kernel decisions attach their id to the active span through the observe hook. Carries enrahitu://022.
Hardening: trusted client identity, the operator role gate, exposure review of every route
X-Forwarded-For is evidence, not identity: it becomes identity only when the operator states how many hops in front of the cell are theirs. This spec adds the client identity resolver the rate limiter keys on, the operator role gate that internal surfaces (the trace ring, ledger inspection, backup triggers) sit behind with no rate limiter on top, and an exposure table that every mounted route must appear in with its authentication class, asserted by a test so an unreviewed public route cannot land. Carries enrahitu://025.
The resource server: bearer tokens rauthy issued, audience bound, scope gated, never app minted
A browser is not the only client. Command line tools, agent runtimes, and MCP clients hold no cookie and cannot follow a redirect into a browser session, and the chassis has so far had nothing to say to them. This spec makes the app an OAuth 2.1 resource server over the same IdP: it publishes protected resource metadata so a client can discover the authorization server, validates rauthy-issued JWT access tokens locally against the cached JWKS with a mandatory audience check, gates routes on scope, and answers an unauthenticated request with the challenge that bootstraps discovery. Registration and the grants themselves stay rauthy's: the app issues no API key, ever.
Streaming responses: server-sent events as the one mechanism, declared per route, bounded and cancellable
A response that is produced over seconds rather than milliseconds breaks every default the edge holds: compression buffers it, the request timeout kills it, the rate limiter counts it once and forgets it, and graceful shutdown drops it mid-sentence. This spec adds one streaming mechanism, server-sent events, and makes a streaming route declare itself so those four defaults are turned off deliberately rather than discovered in production. Streams are bounded by a per-stream channel and a per-client concurrency limit, are cancellation-aware, close cleanly on shutdown, and are one span each in the tracer.
The verbs and the binary composer: preflight, migrate, backup, restore, ledger verify
Operations that exist. rahi-ops implements preflight (config, keys, both stores' health, disk), migrate (spec 011's migrations as a deploy step, leader-only), backup (one encrypted archive holding the app's hiqlite snapshot taken through the addon, rauthy's snapshot taken through rauthy's own API, and the key material that decrypts both), restore (a cluster reset that refuses partial input and is single-shot by a marker), and ledger verify (spec 013 over an exported chain). rahi-cli is the binary composer: an app implements the Cell trait and its main is rahi_cli::run(cell), which yields serve plus every verb with the four exit codes. Carries enrahitu://027.
Single-container packaging: one image, one volume, first boot, keys, die-together supervision
One container plus one volume is a complete authenticated application. The image carries the app binary and the rauthy release binary. First boot generates every key (ledger, session, backup, rauthy ENC_KEYS, the admin token), writes rauthy's environment derived from one public URL, and bootstraps the OIDC client; a restart regenerates nothing. The app is the supervisor: it starts rauthy on loopback, waits for its health, starts serve, and when either exits the other is stopped and the container exits. The volume layout is /data/hiqlite, /data/rauthy, /data/keys, and /data/backups. Carries enrahitu://007.
Cluster topology: N=1 primary, N=3 StatefulSet on Kubernetes, one key set, S3 backups
What multiplies and what does not. At N=1 nothing changes. At N=3 every replica is identical: a StatefulSet pod running the supervisor, with its own volume, two Raft clusters per replica (rauthy's and the app's), peers discovered by stable pod DNS, one key set injected as a Secret and custodied once, the public origin on an Ingress that terminates TLS and keeps /metrics off it, a CronJob that runs backup --to s3:// against the leader, and a ServiceMonitor example. Rate limiting is per node and says so. A shared volume between Raft members is refused by the manifests themselves. Carries enrahitu://030.
The dev substrate and the harness: compose for N=1, boot the real binary, wait on /readyz, a cookie jar
Two things a contributor needs: a compose file that runs the N=1 topology (the image with a bind-mounted volume, watched rebuilds) and a harness crate that boots a built cell binary under a throwaway data directory with fresh keys and OS-allocated ports, starts a rauthy on loopback when one is available, waits on /readyz and never on /healthz, and hands back an HTTP client with a cookie jar so authenticated flows are testable. One instance per test file, never per test. Carries enrahitu://033.
hello-cell: the reference app that composes every crate and proves the chassis end to end
The smallest complete governed cell and the thing consumers copy from: a manifest declaring one table, one kv prefix, and the grants a notes service needs; a Cell implementation whose main is one line; a notes resource with create, list, and delete behind login, each write staged with an outbox row and stamped with a revision, each denial ledgered; a static page that logs in through the proxy and lists notes; an operator route exposing the trace ring; and an end-to-end test through the harness that logs in, writes, reads, is denied an ungranted operation, verifies the ledger, takes a backup, and restores it into a fresh directory.
Denials survive a graceful shutdown, and every lost denial is counted by its cause
Constitution X says the kernel ledgers every denial. Spec 015 answers a denial before its record is durable and drains the records through a bounded queue, and serve never drains that queue on shutdown, so a graceful stop silently loses the denials still queued (reproduced: 50 denials answered 403 with decision ids, 33 and 40 of them in the chain after SIGTERM, nothing counted). A second loss is structural: a decision id is the chain head at boot plus a counter, so replicas of spec 032's StatefulSet that boot on the same head mint the same ids, and the second append of an id is refused as a conflict (reproduced in process: two kernels on one chain both answered with the same id and one record landed). This spec drains the queue within a bound when serve stops, counts what the bound abandons, separates a dropped record from a failed append on /metrics, and names the replica in every decision id, so the only uncounted loss left is a process that is killed.
Manifest and schema evolution: a ledgered manifest transition, a checked migration history, and a restore that refuses what it cannot serve
Today a cell's manifest is frozen at its first boot: the manifest hash is the chain's genesis parent, Ledger::open checks the first record against the booted manifest on every open, and any change to the manifest (one added grant, otel switched on) makes serve exit 1 with an integrity error on the existing volume. Spec 015 B-8 names the missing step, "a deploy genesis record", and nothing defines it. Migrations have the mirror problems: serve accepts a store ahead of the binary, a changed migration under an applied version is skipped silently, and restore checks no compatibility. This spec makes a manifest change a signed transition record appended at deploy, checks the booted manifest against the chain's current manifest, records a checksum per migration, lets an older binary serve a newer schema only across migrations declared additive, and makes restore refuse an archive it cannot serve. It carries the worked consumer example for an upgrade, a rollback, and a restore.
Identity recovery and the live proof: a backup a real rauthy answers, a restore that brings the users back, and CI that runs against the pinned release
Recovery of a cell is two state systems, and only one of them recovers today. Against a real rauthy the backup verb fails outright, because rauthy's backup routes accept only an admin session and the verb presents the admin API key (030 D-3, observed against 0.36.2). The app half races hiqlite's background backup and can report no file for a backup that lands a moment later (observed on two of four runs). A restore places rauthy's snapshot and nothing hands it to rauthy, so a restored cell comes up with no users, and every sub in its rows is unreachable. CI never runs a real rauthy. This spec resolves D-3 by the mechanism a human chooses, waits for the app backup to exist, applies rauthy's snapshot exactly once on the next start, and adds a CI job that runs every rauthy-gated test, and an end-to-end test through recovery with identity, against the pinned rauthy release.
Native clients and bearer revocation: the CSRF exemption wired, a public client for a CLI declared in the manifest, token lifetimes set, and a token that can be revoked
A hosted control plane built as a cell and a command-line client that logs in to it need four things the chassis does not yet give. The CSRF layer exempts only /auth/*, so a POST with a bearer token is refused 403 csrf (025 B-11 promised the exemption; D-6 left the wiring). No public client exists: the one client first boot creates is confidential, and a rauthy client defaults to EdDSA with no audience binding, so no CLI token passes the resource server without an admin's hand. Token lifetime is rauthy's default, 1800 seconds, not the fifteen minutes 025 B-5 states. And nothing writes the jti deny-list, so a bearer token cannot be revoked before it expires. This spec wires the exemption, declares native clients in the manifest and provisions them at boot bound to RS256 and the cell's audience, sets lifetimes explicitly, adds revocation by token and by subject, reports the bound in preflight, and gives hello-cell a scope-gated bearer route and a device-code login in its end-to-end test.
Releases and out-of-tree packaging: a tagged version a consumer pins, images that exist under the names the manifests use, and a runtime image a cell in another repository builds on
The thesis says consumers depend on published crates and pin a version (002 D5), and nothing is published: no tag, no release, no crate, no image. The one route that works is a git dependency pinned to a sha. cargo package fails on rahi-ops because it includes a template from outside the crate, which also blocks rahi-cli, the crate that carries Cell. The deploy manifests name ghcr.io/bartekus/rahi:latest, a namespace and a tag no workflow produces. The image recipe builds only a package of this workspace, and a cell's static directory named under CARGO_MANIFEST_DIR does not exist in the image (hello-cell's page answers 404 there). This spec defines the version and the release, publishes the images a tag builds under the names the manifests use, adds a runtime image an out-of-tree cell builds on, makes static assets exist in the image, makes the crates packageable, and proves the consumer stanza on every release.
Runtime identity: a replica states which bits it runs, which ceiling it enforces, and which instance it is, with every value's basis named, on bounded surfaces
A consumer that correlates an immutable artifact, an authority snapshot, a deployment, and a running replica needs the replica to say what it is. Today it says almost nothing: rahi version prints the crate version, the manifest hash is visible only inside a ledger export or a backup, the OTel resource carries service.name alone, the image has no labels, and the deploy manifests pin no digest. This spec makes a replica report its measured executable digest, its booted manifest hash, its declared image and build revision, and a per-boot instance id, each with its basis (measured, declared, or absent), on one unguarded document at /binding kept off the ingress like /metrics, on the OTel resource, and in one build-info metric whose labels are versions only. It states what the telemetry does not observe or count. It writes nothing to the chain; deployment epochs are spec 041.
Deployment epochs: the chain records each deployment it serves under, appended at the deploy step, linked to external build and deployment records it never rewrites
The chain commits to the first manifest (genesis) and, with spec 036, to every manifest adopted since. It records nothing about which build a replica runs or which deployment put it there, so a consumer cannot join a decision to an artifact, an authority snapshot, or a rollout, and an upgrade, a rollback, and a restore look the same in the chain. This spec appends one epoch record at each deploy step that changes what is deployed: the manifest current after the step, the artifact (the binary measured by the deploy step, the image digest the deployer declares), and typed opaque references to the external build provenance, the deployment record, and the authority snapshot. The genesis and every existing record stay byte for byte; no history is rehashed; nothing is embedded in a built artifact; every reference points from the later record to the earlier one, so no hash cycle is possible. Replicas bind to the current epoch at boot and report a mismatch without refusing, every decision names its epoch and instance, and the chain becomes readable from a running cell.
Lifetime identity: one id names one decision for the life of the chain, and absence is proven rather than assumed
The chain can say what happened but not whether a given decision ever happened, because spec 013 D-5 tells a lost compare-and-swap from a duplicate id by asking whether the id is resident and spec 014 deletes the row when its segment is sealed. Past the hot window a retried append therefore writes a second record under the same id, and full verification passes because a duplicate id is not a broken link. This spec gives every decision a narrow resident identity row that survives sealing, makes that row's primary key the arbitration inside the append transaction rather than a read before it, stamps the row in the seal transaction, and answers presence, proven absence, unproven history and proven ambiguity as four different answers. A chain sealed before this spec has no rows for its archived ids, so a cell refuses to serve until `rahi ledger reindex` has rebuilt them: the upgrade costs a stop, and the alternative is a cell that serves while its audit proof is silently wrong. The refusal, the reindex's behavior on damaged or ambiguous history, the cutover from a binary that does not stamp, and the permanent resident cost are all stated here rather than discovered in production.
Patched dependency adoption: the N=1 cell runs the published, provenance-pinned hiqlite and Rauthy builds, crosses their cache boundary only through a fenced, resumable, backed-up transition, keeps every accepted revocation through it, and stops within a validated budget with an honest outcome
rahi's application store runs hiqlite 0.14.0 through a root git patch that registry consumers never inherit, and its co-deployed identity provider runs upstream Rauthy 0.36.2. Published downstream builds exist for both, hiqlite-patched 0.15.0-patched.1 and rauthy-patched 0.36.2-patched.2, each qualified by its producer at N=1. This spec adopts exactly those two artifacts for the N=1 cell under a narrow recorded exception to 011's no-fork rule. It crosses the one incompatible on-disk boundary (the cache raft log) by relocating the app store behind a permanent fence that every pre-043 binary refuses at its old path, installed whole and verified quiescent before anything moves, and a second fence at rahi's rendered Rauthy environment that makes every published pre-043 supervisor that reads it afterwards exit before it spawns Rauthy; it names what neither fence can stop (an old process already past its read or its lock-free start phase, and the two destructive hiqlite variables in an old process's environment) as preconditions the operator establishes by stopping and removing every old process first. Every new-binary entry point takes its locks before it reads the state file; Rauthy moves its own cache under consent, and the resumable upgrade is released only on a Rauthy build whose interrupted move is repaired. It keeps every revocation accepted before the upgrade refused by a permanent issued-at floor that needs no historical lifetime, bounds every admitted token by an enforced lifetime ceiling and a hard 86,400-second maximum, keeps durable revocations against a backward clock step after any prune (retained, or pruned under a persisted watermark, by owner choice), states the restore boundary, including what a stale Rauthy restore revives, turns a proven terminal storage failure into a bounded exit, and reports every stop's observed outcome with a non-zero exit when completion is not confirmed. It changes no topology and claims no N=3 support.
N=3 split composition: two StatefulSets, rahi and rauthy, one hiqlite node per pod, rahi reaching rauthy only through an encrypted internal Service and losing readiness, never liveness, without it
The owner decided on 2026-09-23 (hiqlite proposal D-14, recorded at bartekus/hiqlite@da4c910) that an N=3 cell is one namespace holding two StatefulSets, rahi and rauthy, three replicas each, one hiqlite node per pod, and that the single-container supervised composition of spec 031 stays the N=1 profile. This spec is rahi's share of that decision. It amends 031 and 032 for N=3 only: rahi starts without rauthy, reaches it only through the internal ClusterIP Service rauthy-internal.<namespace>.svc.cluster.local over Rauthy's TLS or a mesh's mTLS, fails readiness and never liveness while rauthy is unavailable, and keeps peer discovery independent of readiness. It carries the proposal's routing, NetworkPolicy, per-StatefulSet disruption and storage requirements, and backup obligations. It changes no dependency and claims no N=3 support: that follows only from recorded qualification on a named release.
Receipts and work claims: an inbound item is recorded once per content revision, processed once per processor revision, and never silently dropped
Every product that ingests from an outside source rebuilds the same table: a key, a body digest, the first outcome, and a rule for a repeat (statecraft-platform's `sc_idempotency`; aicortex's work-claim rows over the Lease and the Outbox). This spec puts one primitive in rahi-store. A receipt names an inbound item by tenant, source namespace, and external key (or a synthetic key when the source has none) and records a digest of its content; classification distinguishes first seen, exact redelivery, changed content under the same key (recorded as a new revision or a collision, never dropped), and erased. Receipt rows stage into the caller's TxnBuilder beside the domain writes and Outbox::stage. Processing is identified separately (receipt revision plus processor and its policy revision) so a correction or a new policy reprocesses. Work is reserved under the existing Lease into a fenced claim row with attempt history, bounded retry, a visible dead-letter state, and reclaim after expiry; queue counts reach /metrics redacted. Retention compacts to a tombstone, erasure keeps only the key digest and is never undone, and a crash-point table states recovery at every boundary. Owner decisions of 2026-09-24 are recorded.
Named migration sets: a linked library brings its own versioned, checksummed migration sequence, and never renumbers into the host's
A store today has one migration sequence, owned by the cell. A library linked into a cell (aicortex inside travel-memory) numbers its own migrations from 1, and so do the chassis's own tables (coordination_migration, 045's receipts), so the host must renumber everyone into its list and every renumbering is a checksum nobody else can reproduce. This spec gives each linked library a named migration set: the store records version, checksum, and the additive declaration per (set, version); migrate plans across sets with declared minimum-version requirements between them; serve, backup, and restore apply 036's checksum-first and additive rules per set. A store written before this spec keeps its schema_version rows byte for byte as the implicit set app. 045's receipt_migration becomes the chassis set rahi.receipts.
Manage application background services through the cell lifecycle
A Cell can declare routes and migrations, but it cannot give the chassis an application worker whose lifetime is tied to serve. Applications therefore either omit durable background work or detach tasks that Rahi cannot cancel, observe, or join before the application store shuts down. This spec adds a named managed-service declaration to Cell, starts each service once after successful composition, treats an unexpected return, error, or panic as a process failure, broadcasts cancellation for every stop cause, and joins or aborts every service before draining its final denials and shutting down the store. The service bound overlaps the existing HTTP drains so the published stop grace does not grow.
Make every exit of a program that opened the app node pass through its shutdown
hiqlite creates an unclean-stop marker, state_machine/lock, when a node starts and removes it only in a graceful shutdown; without the auto-heal feature the next start refuses until someone repairs the directory by hand. The travel-memory service hit this with a ? between Store::open and Store::shutdown and a Ctrl-C-only handler. This spec gives rahi-store one scope for the node's lifetime, Store::run, which arms SIGTERM and SIGINT before the node starts and shuts it down on every way out of the body; it closes the three paths in rahi's own verbs that returned with the node open and the signal window in serve's boot; it documents the single-voter recovery and verifies it; and it records why auto-heal stays off after a probe showed its rebuild complete.