Skip to content

Rules promotion: optional evidence gate before rules approve, and integrity check of the managed block #1427

Open

Description

@belchiordameao-cmd

Context

docs/design-rules-promotion.md makes promotion a human command (§6) and keeps the block small by tests and budget (§4–5). Two things it does not cover, which bit us in a field setup that runs the same idea by hand (a small always-loaded memory index shared by ~15 concurrent agent sessions, regenerated from a larger note store):

  1. Approval is judged on the text, not on behaviour. A regenerated always-on file passed a 14/14 holdout and still regressed a domain the holdout did not cover (one financial fact inverted). It was live for ~1 minute before a manual rollback.
  2. Nothing watches the installed file. Another writer, a truncation or a hand edit changes what every session loads, and nobody notices until behaviour drifts.

[auto_improve.eval] already gates auto-improve proposals with a scorer and min_delta. This asks for a paired, non-inferiority version of that idea at rules approve time, where one regression in a high-stakes fact must block even if the mean score holds.

Proposal (both opt-in, no change to the default flow)

A. rules approve --gate (or a [rules_promotion] gate config): before writing the block, run a fixed case set against the current block (A) and the candidate block (B), each case with an objective expected answer (regex on fact/value/tool). Decide by paired non-inferiority, not by raw score:

  • pair results per case; majority over k runs per arm (tie = worse);
  • exact McNemar on the discordant pairs, Clopper-Pearson interval scaled by (b+c)/N, margin −2 pp;
  • hard veto: any new error on a case tagged class: a (facts where one wrong answer causes immediate damage: payment details, contact numbers, prices) blocks, whatever the interval says.
    The file that was tested is the file that gets installed (byte compare), so the gate cannot pass one artifact and install another.

B. Managed-block integrity: approve/edit/remove record the sha256 of the block they wrote. A rules verify (cron/hook friendly) compares; on mismatch it logs on the first pass and restores the last approved block on the second consecutive pass; empty, over-budget or gate-failing content restores immediately. One command rolls back to the previous approved version.

Numbers from our setup (aggregated)

  • Synthetic check of the gate, N=129: identical → promote; 1 worse case → promote (CI [−0.008, +0.007]); 3 worse → reject (CI [−0.023, +0.006]); 1 worse in class a → reject.
  • End-to-end: a candidate with 4 swapped facts was rejected in <1 min; the installed file's sha was unchanged.
  • Integrity watcher, 7 states tested: trigger→restore in ~112 ms; worst case via a 5-min schedule ~4 min.
  • Lesson on B: a legitimate second writer that rewrites the file without updating the recorded sha produces false alarms; the writer has to go through the same command.

Known limits

  • A small case set (15 cases, 7 class a) catches gross and class-a regressions, not fine ones; cases drawn from the training notes detect, they do not generalise.
  • If the gate's runner cannot run (expired credential), we fail closed, which blocks every new approval until someone sees the alert.

Question
Does an opt-in gate like A fit §6 (command-driven, no auto-apply), as a check the command runs rather than a background writer? And does B belong in core or in a companion?

Activity

  1. added
    rfcOpen design discussion / request for comments — excluded from automated iss-audit triage
    on Oct 19, 2026
  2. akitaonrails commented on Oct 19, 2026

    @akitaonrails
    Owner

    Audit result: this is a design proposal, not a defect in shipped behavior. The referenced rules approve flow is explicitly marked not implemented in docs/design-rules-promotion.md, so adding an evaluation gate or integrity restorer before that base workflow would create policy and persistence surface out of order. Keeping this open as an RFC. The rules-promotion design should first settle the canonical managed-block owner, artifact identity, rollback storage, and command authorization; the optional non-inferiority gate and integrity verification can then be evaluated as part of that design rather than landed independently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    rfcOpen design discussion / request for comments — excluded from automated iss-audit triage

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions