sanitize-* policies never redact: they build a [REDACTED] string nothing reads, and mislabel a non-blocking event as "Blocked" #669

Description

The five sanitize-* policies (sanitize-api-keys, sanitize-jwt,
sanitize-bearer-tokens, sanitize-connection-strings,
sanitize-private-key-content) present themselves as redaction: name, description,
displayTitle, impact and message all promise it. Each returns, in
src/hooks/builtin-policies.ts (~L594 on 1.0.0-beta.12, ~L373-384 on 0.0.15, for
sanitize-api-keys):

return {
  decision: "deny",
  reason: `${label} detected in tool output`,
  message: `[REDACTED: ${label} removed by failproofai]`,
};

No redaction happens. Three distinct problems compound on this path, listed here so
they can be triaged separately:

1. The redaction never reaches the model. All five policies match
   { events: ["PostToolUse"] }. On Claude Code a deny on PostToolUse changes
   nothing — the tool has already run and its output has already reached the model — and
   failproofai itself is correct about this: ENFORCEMENT_CAPABILITY
   (src/hooks/enforcement-capability.ts) classes PostToolUse as observe, because a
   deny there is inert. So on this event the policy result becomes
   hookSpecificOutput.additionalContext and nothing else. Meanwhile Claude Code exposes
   a redaction channel on the very same event that is never used (see below).

2. PolicyResult.message — the field carrying [REDACTED: …] — is dead code. It has
   no reader anywhere in src/; the PostToolUse branch of policy-evaluator.ts reads
   only reason. The string is constructed and discarded.

3. The injected text asserts a block that did not happen. The additionalContext
   actually sent to the model is:

   Blocked Bash by failproofai because: AWS access key ID detected in tool output
   

   Nothing was blocked — the tool ran, the output (secret included) reached the model —
   and telling the model "Blocked" on an event that cannot block is misleading. This is
   independent of Bump eslint from 10.1.0 to 10.2.0 #1 and Bump next from 16.2.1 to 16.2.2 #2 and fixable on its own.

The net effect is a detector dressed as a redactor: a secret read by a tool reaches
the model and lands in the workstation's transcript; the policy then tells the model,
after the fact, that it was "blocked".

The registry metadata states the promise in as many words. On 1.0.0-beta.12
(builtin-policies.ts:2040-2096) the five policies carry, alongside their PostToolUse
match:

displayTitle: "Redacted API keys from tool output",
impact: "Catches OpenAI / Anthropic / GitHub / AWS / Stripe / Google keys before the model sees them.",

and, for sanitize-connection-strings, "Strips embedded DB credentials before they
reach the model context." On a PostToolUse hook, "before the model sees them" is not
what happens: the tool ran and its output was delivered before the policy was consulted.
These strings are what a user reads when choosing which policies to enable.

Steps to Reproduce

Environment: policies installed with
failproofai policies --install <names> --cli claude --scope user, then one real
PostToolUse event whose tool output carries a secret:

printf '%s' '{"session_id":"x","transcript_path":"/tmp/t.jsonl","cwd":"/tmp",
  "hook_event_name":"PostToolUse","tool_name":"Bash",
  "tool_response":{"stdout":"AKIAIOSFODNN7EXAMPLE","stderr":"","interrupted":false}}' \
  | failproofai --hook PostToolUse

Observed (failproofai 0.0.15, real binary):

{"hookSpecificOutput":{"hookEventName":"PostToolUse","additionalContext":
 "Blocked Bash by failproofai because: AWS access key ID detected in tool output, …"}}
exit=0

Exit 0, no decision field, no [REDACTED] anywhere, and the "Blocked" text
described in #3. The secret is unchanged in what the model receives and in the JSONL
transcript.

Expected Behavior

The redaction the policies advertise is technically reachable on this event — Claude
Code 2.1.223 declares on PostToolUse:

updatedToolOutput: E.unknown().describe("Replaces the tool output before it is sent to the model")

and it has a real consumer (He = Ve.updatedToolOutput then await et(He, Ve)). Verified
end-to-end with a witness hook on both Bash and Read: the model received
[REDACTED: AWS access key ID] and said so, and the JSONL transcript no longer contained
the secret.

This is the same channel your own ENFORCEMENT_CAPABILITY table already names: the
copilot PostToolUse entry reads "Post-hoc anyway (result-rewrite only)". So the ask
is not "make deny work on PostToolUse" (it can't, and your table is right to say so); it
is use the result-rewrite channel you already document:

1. Emit hookSpecificOutput.updatedToolOutput from the five sanitize-* policies, so
   the redacted form actually replaces the tool output. One caveat we hit and that shapes
   the ask: Claude Code validates updatedToolOutput against each tool's output
   schema, and a shape mismatch is fail-open — the attachment reads
   PostToolUse hook returned updatedToolOutput that does not match Read's output shape; using original output, i.e. the secret passes through. So the redactor must know
   and track the per-tool result shape (Bash → {stdout,stderr,…}, Read →
   {type,file:{content,…}}, MCP tools → their own), not just substitute a string. The
   failure is at least noisy (visible attachment), not silent.

2. Resolve the dead message field — either wire PolicyResult.message into the
   redaction path (it already carries the intended [REDACTED: …]) or remove it, so the
   type stops advertising a capability nothing consumes.

3. Stop saying "Blocked" on a non-blocking event. The additionalContext text for
   observe-class results should describe what actually happened (a detection), not a
   block. This is the cheapest of the three and worth doing regardless of Bump eslint from 10.1.0 to 10.2.0 #1. The same
   applies to the displayTitle / impact strings quoted above: until Bump eslint from 10.1.0 to 10.2.0 #1 lands, they
   describe a capability the code does not have, and they are what a user reads when
   deciding whether they need another control.

These are one coherent change (finish the redaction path the sanitize-* policies imply)
but can land independently; #3 alone already removes a misleading signal.

Version

0.0.15 (current npm latest, and the version we run): the sanitize-* behaviour above
was reproduced there against the real binary. Also checked against 1.0.0-beta.12
(npm next, published 2026-08-07): updatedToolOutput appears nowhere in src/ or
dist/, the five policies still match PostToolUse with defaultEnabled: true, and the
unread message: "[REDACTED: …]" is still built (builtin-policies.ts:580-651). Nothing
here is fixed on the 1.0 line. Point #1 hinges on Claude Code exposing
updatedToolOutput, verified on Claude Code 2.1.223.

Node.js / Bun Version

Node 22

Operating System

macOS

Additional Context

We run failproofai in a hardened fleet configuration at a bank, alongside an OS-level
sandbox, and we surfaced this while auditing whether the sanitize-* family is a genuine
DLP control (it is currently listed as one in our own security-requirements review — which
we have since corrected). We keep all five policies enabled as a leak signal, not a
control: measured on 4267 tool results across 231 local sessions, they fire 3 times
— cost is effectively zero, so there is nothing to lose by keeping them, and the signal has
value the day the redaction lands.

Two adjacent notes in case they help scope the fix:

• Your ENFORCEMENT_CAPABILITY table is not in the wrong here — it answers "does a
  deny change behaviour?", and a deny on PostToolUse genuinely doesn't. The redaction
  channel is a different mechanism, and the table already gestures at it on the copilot
  row ("result-rewrite only"), so Bump eslint from 10.1.0 to 10.2.0 #1 is additive, not a contradiction of your own docs.
  What the table does not do is stop the sanitize-* metadata from promising a redaction
  that the observe classification rules out.
• Happy to supply the witness hook that proves the updatedToolOutput round-trip
  (model + transcript) if useful.