Skip to content

ci(publish-smoke): a cancelled smoke reports pending "nothing measured", never failure - #24503

Merged
os-zhuang merged 1 commit into
mainfrom
claude/issue-24499-publish-smoke-cancelled-not-failure
Oct 19, 2026
Merged

os-zhuang merged 1 commit into
mainfrom
claude/issue-24499-publish-smoke-cancelled-not-failure

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #24499
Clause-②: no

What changed

One file, .github/workflows/publish-smoke.yml (+14 / -5, 19 changed lines):

  • The Report verdict to the release PR head step now maps job.status with a case. cancelled writes pending with the description "Cancelled before a verdict (superseded by a newer smoke, timed out, or stopped by hand) — nothing measured" (106 characters; the status API caps descriptions at 140). success and failure write exactly what they wrote before. Any other value still falls through to failure.
  • Three header lines say how to read the new entry when you walk the status history: a pending reading "Cancelled before a verdict" closes a run that measured nothing.

The step's if: (always() && needs.resolve.outputs.report-sha != ''), the Mark pending step, resolve-guard, the concurrency block and every other job are untouched.

Mapping, before and after

job.status in the verdict step when it is reached before (state · description) after (state · description)
success every step passed success · Fresh install of the release candidate: auth + CRUD green unchanged
failure a step failed (build, pack, install or a probe) failure · Release candidate fails a fresh install — see the run log unchanged
cancelled cancel-in-progress supersession (all 75 measured cancellations), a job timeout-minutes overrun (none measured), or a manual stop failure · Release candidate fails a fresh install — see the run log pending · Cancelled before a verdict (superseded by a newer smoke, timed out, or stopped by hand) — nothing measured
skipped not reachable. skipped is not a job.status value, and a skipped pack-smoke job runs no steps, so nothing is written before or after this change — —
any other value none is documented failure failure (the *) arm keeps the fail-closed default)

How I checked the table: I extracted the step's run: body from the YAML, substituted each job.status value, and ran it under bash with a stub gh that prints the state= and description= it receives. I did this at BASE 959ee31 and at head 0c5ad58. The table above is that output. The probe script lives in my scratchpad; no test file is added.

Mechanism, measured (each PM hypothesis)

  1. The step at origin/main 959ee31. Confirmed as the dispatch described it. The verdict step is at :236–:249: if: always() && needs.resolve.outputs.report-sha != '', and any job.status other than success became state=failure with "Release candidate fails a fresh install — see the run log". The concurrency block is at :73–:75: group: publish-smoke-${{ github.event_name }}-${{ github.ref }} with cancel-in-progress: true.
  2. What happens on a cancel.
    • GitHub's docs (github/docs at dae83f7d28, content/actions/reference/workflows-and-actions/contexts.md) say: "job.status … Possible values are success, failure, or cancelled". expressions.md says always() "Causes the step to always execute … even when canceled". workflow-syntax.md says jobs.JOB_ID.timeout-minutes is "The maximum number of minutes to let a job run before GitHub automatically cancels it", so a job timeout is also a cancelled.
    • The REST job and step records of the two runs named on the card:
      • run 39221870768, job 117958850974: Build is cancelled, Publish smoke (packed tarballs) is skipped, and Report verdict to the release PR head ran and concluded success. The job is cancelled.
      • run 39222297525, job 117960447073: the same pattern.
    • The status history on 4ed16392ee shows both runs writing failure (08:22:52Z and 08:24:32Z). So the always() step does run on a cancel, and job.status there is not success. No step failed, so of the three documented values it read cancelled.
    • ⛔ NOT MEASURED directly: the literal rendered value. The job logs redirect to a blob host that this container's proxy refuses (403), so I could not read the expanded script line.
    • Ordering. The superseding run's first job starts only after the cancelled run has ended. Run 39221870768 ended at 08:22:55Z and the Resolve target job of 39222297525 started at 08:22:56Z. Run 39222297525 ended at 08:24:36Z and Resolve target of 39222454134 started at 08:24:37Z. So a cancelled run's write always lands before the newer run's own pending.
  3. Route: (b) with pending. The four-axis reasoning is in the next section.
  4. Parsers of this workflow. All are green, listed under Gates. check:workflow-status-functions scopes job-level if: only, and this diff changes no if:.

Route choice — (b) pending + "nothing measured"; not (a) silence, not error

Census. I listed the publish-smoke.yml runs through the REST run list (last 300, read 2026-10-19 around 08:40Z). Of these, 299 are workflow_run smokes created between 2026-10-16T00:25Z and 2026-10-19T08:23Z: 224 success, 75 cancelled, 0 failure. For every one of the 75 cancelled runs, a newer smoke was created before it ended. The longest cancelled run lasted 9.1 minutes against a 45-minute job timeout. So no timeout and no stand-alone cancel shows up in the window.

The options and their failure modes.

  • (a) Write nothing on a cancel. No red, and the newer run's pending follows. But the cancelled run's last entry stays "Installing the release candidate into a fresh project…", which stops being true the moment the run is cancelled. Someone walking the history then has to infer "superseded" from two pending entries in a row. On a timeout or a manual stop with no later smoke, the head would show a false "Installing…" indefinitely.
  • (b) error + "nothing measured". Never leaves a stale pending. However, error renders red, just like failure, and the combined status rolls it up as a failure. The window between the cancelled run's write and the newer run's pending is about 15 seconds both before and after this change (08:22:52Z → 08:23:08Z, 08:24:32Z → 08:24:46Z). So error keeps the exact red flash the maintainer escalated, on about a quarter of all smokes. Only the wording would change.
  • (b) pending + "nothing measured" (chosen). No red on a supersession. The cancelled run closes its own entry in words, so the history explains itself. Its stale case is bounded:
    • It needs a cancel that is not a supersession (timeout or manual stop, 0 of 75 measured) and also no later Release run before the head merges. Every push to main starts a Release run, and that run starts a new smoke on the open release PR.
    • Even then the head reads an honest pending that says "nothing measured". That is never green, and it never claims a measurement.

Four axes.

  • 实际业务需求 (real business need). The scenario that actually occurs is supersession: 75 of 299 smokes, every cancel in the window. That is the case the maintainer hit on chore: version packages (next) #24477. A timeout or manual cancel has zero measured occurrences, so the route is chosen for the observed case and only kept honest for the others.
  • 项目长远合理性 (long-term soundness). Every run now closes its own entry in the status history with a truthful state, which matches "absence must be loud". The fix stays one case with no new job and no extra API call. I rejected discriminating supersession from a timeout at run time (for example by looking up newer runs during the cancel): it adds machinery and new failure modes, and nothing measured asks for it.
  • 防 AI 写错 (keep AI from getting it wrong). This card was itself filed from a history walk that read failure entries. With the change, failure stays reserved for a smoke that ran and failed (plus resolve-guard's self-describing "never smoked", unchanged). A cancelled run says so in words, and the combined status never reads red for a run that measured nothing. With error, a rollup reader would see red again; with silence, a reader has to know an inference rule.
  • 创业阶段不扩散 (no scope sprawl at this stage). No new gate, job or required check, and one file. A cancelled run makes one extra status write, about 75 over 3.5 days.

Wanted / Done when — conclusions

  • Wanted 1. A smoke that did not reach a verdict writes no verdict. Done: cancelled maps to pending "Cancelled before a verdict … — nothing measured" and never to failure.
  • Wanted 2. A genuinely failing smoke keeps writing failure exactly as today, and resolve-guard's explicit red stays. Done: the failure arm is byte-identical in state and description, and resolve-guard is not in the diff.
  • Wanted 3. No new gate, no required-check change, Tier H draft. Done: the PR is a draft and stays one for an authorised approval.
  • Done when 1. See the mapping table above (skipped is covered as unreachable).
  • Done when 2. The PR names the expected reading of the next superseded smoke (this paragraph) and the seat records the first reading on main after the merge.
    • The PR's own CI cannot exercise this. A workflow_run workflow runs the file on the default branch, and the dispatch rules out dispatching it. The first reading therefore comes after the merge.
    • Expected status history on the release PR head:
      1. run A: pending "Installing the release candidate into a fresh project…";
      2. run A: pending "Cancelled before a verdict (superseded by a newer smoke, timed out, or stopped by hand) — nothing measured", linking run A, which concludes cancelled;
      3. run B: pending "Installing…";
      4. run B: success or failure.
    • No failure "fails a fresh install" may come from a run that concluded cancelled. The seat records the first such reading (run id and status history).
  • Done when 3. Run 39222454134, in flight at filing, concluded success (REST run record), so there is no candidate failure to split off and no separate card.

Acceptance notes

  • Timeout loudness changes direction. A job that overruns its 45-minute timeout-minutes is a cancelled job (docs above). It used to read red failure "fails a fresh install" and now reads pending "Cancelled before a verdict … timed out …". No such run appears among the 299 measured. Making a hung smoke read red would be separate work; carrier: none (承接者:无).
  • Bounded stale pending. If a cancel is not followed by any later smoke before the release PR merges (a timeout or manual stop, then no push to main), the head keeps the honest pending above. It is not red and not green. A re-run of the smoke or the next Release run replaces it.
  • origin/main moved after BASE. It advanced to 7e58e0a (a packages/cli test budget only), which is disjoint from this file, so I did not merge it. The PR's merge ref and the queue cover it.

Gates

Derivation: node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack, run with no paths. It read the change set from git against merge base 959ee31: 1 path, 19 changed lines. It listed 39 commands: 31 matched by path and 8 that run on every change.

Local run at head 0c5ad58 (git rev-parse --short HEAD). Every command's output went to its own log, and the exit code was captured before any pipe. All 39 exited 0:

  • Direct node scripts (17):
    • check-aggregator-roster.mjs and --self-test
    • check-closing-keyword-parity.mjs and --self-test
    • check-comment-mask-corpus.mjs
    • check-position-name-fold-loaders.mjs and --self-test
    • check-self-test-wired.mjs and --self-test
    • check-self-test-workflow-commands.mjs and --self-test
    • check-step-collectors.mjs and --self-test
    • check-whole-set-label-write.mjs and --self-test
    • docs-audit/check-drift-comment.mjs
    • pm/ci-failure.mjs --self-test
  • pnpm gates (22):
    • check:agent-test-spelling, check:cross-package-test-inputs, check:declared-population-live, check:driver-memory-census
    • check:gitlink-declared, check:node-version, check:nul-bytes, check:pm-dispatch-gates (fast tier, 1913 cases)
    • check:pm-expected-skips, check:pm-governed-merges, check:pm-post-stamped, check:pm-record-recognisers
    • check:pnpm-acquisition, check:pnpm-filter-targets, check:refd-timer-probe, check:required-contexts
    • check:shard-attestation, check:stall-guard-budget, check:stall-guard-headroom, check:watch-hint-literal
    • check:workflow-status-functions, check:workflow-step-name-quoting

Verdict lines:

  • check-workflow-status-functions: OK (scanned 40 workflow file(s), 75 job(s), 32 job-level if: expression(s); 11 read needs.*.outputs.*, all naming a status function).
  • check-workflow-step-name-quoting: OK (scanned 40 workflow file(s) + 2 composite action file(s), 841 step name(s) …)

Reconciliation: --ran with command :: exit N lines printed ✓ dispatch-gates --ran: 39 derived famil(ies) accounted for — 39 run, 0 NOT-MEASURED (a DERIVED zero — all 39 recorded an exit code and none of them is 3).

NOT MEASURED, by value: six families take their argv or env: from a CI run and have no local invocation. They are check-required-contexts.mjs --verify-required-set and five check-shard-attestation.mjs --emit/--verify forms. CI owns them, along with the full farm.

No package is touched, so no build closure and no package tests were needed. No changeset: a workflow file publishes nothing, so the PR carries skip-changeset.

维护者速读(草稿)

改了什么:发布冒烟(publish-smoke)被新一轮冒烟取代而取消时,不再往 Version Packages PR 上写红色 failure「候选版本全新安装失败」,改为写黄色 pending「在得出结论前被取消 —— 什么都没测」。真正失败的冒烟照旧写 failure,成功照旧写 success。只改了一个文件,19 行。

为什么改:#24477 上那个让人以为「ci 崩了」的红叉,就是被取消的冒烟误报出来的。每次合并到 main 都会触发新一轮 Release 和新一轮冒烟,旧的那轮随即被取消。10/16 以来的 299 次冒烟中有 75 次是这样被取消的,真失败 0 次。

风险与代价(含回滚):

  • 冒烟超时(45 分钟)或被手动取消、且之后再没有新冒烟时,PR 上会一直停在黄色 pending(文字如实写明「什么都没测」),不会变红。实测 299 次中超时 0 次。
  • 本 PR 自己的 CI 跑不到这段逻辑(workflow_run 只读 main 上的 workflow 文件),要等合并后第一次被取代的冒烟才能看到效果,席位会记录那次读数。
  • 回滚:revert 这一个提交即可。

席位意见:

你要做的:审阅并批准(.github/workflows/** 属 Tier H,需要你的批准)。合并后无需其他操作。


Generated by Claude Code

…d", never failure

The verdict step mapped every non-success job.status to `failure` with
"Release candidate fails a fresh install". job.status is success, failure
or cancelled, and a cancelled smoke (superseded by cancel-in-progress when
a newer Release run starts, a job timeout, or a manual stop) never reached
a verdict, so that red was false and it polluted the status history the
header tells a reader to walk.

Map cancelled to `pending` with a description saying the run was cancelled
before a verdict and measured nothing. success and failure are unchanged,
and any other value still falls through to failure.

Claude-Session: https://claude.ai/code/session_01FM29ZqLTrnPNXxijioZwMp
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet objectstack-fleet Bot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Oct 19, 2026
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

维护者速读

改了什么:publish-smoke.yml 里「把结果报回 release PR」这一步的状态映射。原来只要不是成功就一律写 failure;现在被取消的运行改写 pending,描述为「Cancelled before a verdict … — nothing measured」。成功和真失败的写法一字不变。只改这一个文件,+14/−5。

为什么改:你说的「24477 ci 崩了」就是这个原因。每合入一次 main,就会触发一次新的 Release,新的 smoke 会把还在跑的旧 smoke 取消掉;旧运行却往 Version Packages PR 上写一条「发版候选全新安装失败」的红。开发统计了 10-16 以来的 299 次 smoke:224 次成功,75 次被取消,0 次真失败。也就是说,这些红全是假的。今天 08:22Z 和 08:24Z 那两条红都属于这种情况,08:32Z 下一次 smoke 写回了绿。

风险与代价(含回滚):真失败照旧报红,resolve-guard 的显式红不动,并发配置和其它 job 都不动,也不加门禁、不改必需检查。有一个已知的边界:如果一次 smoke 超时或被人手动停掉,之后又没有新的 smoke,那条 status 会一直停在 pending,不会变红。实测 75 次取消里,没有一次属于这种情况。回滚就是 revert。

席位意见:建议批准。改动点正好是产生假红的那一步,case 的兜底分支在遇到未知值时依然报红。39 个本地门禁全部 exit 0。

你要做的:给本 PR 一次批准(APPROVED review),之后由 devx 席位落地。

席位核对

ACCEPT · domain:devx seat 1 · os-warren · session_01FM29ZqLTrnPNXxijioZwMp · 2026-10-19T08:58Z · head 0c5ad5810f · card #24499 (claim 6263021771)

  • Form. Draft, base main, first line Fixes #24499 (the only closing keyword), Clause-②: no. Assignee os-warren. skip-changeset: workflow only.
  • Scope. 1 file, .github/workflows/publish-smoke.yml, +14 / −5: the Report verdict to the release PR head step's if/else becomes a case (success → success, cancelled → pending "Cancelled before a verdict … — nothing measured", * → failure), plus 3 header lines on reading the status history. if: always() …, concurrency, resolve-guard and the other jobs are untouched (read in the diff).
  • Mapping, proven by the dev's probe (the step's own run: body extracted and executed under bash -euo pipefail with a stub gh, at BASE 959ee317b and at HEAD): success and failure are byte-identical before and after; cancelled goes from failure to pending.
  • Premise, measured.
    • The two failure writes on chore: version packages (next) #24477's head 4ed16392ee came from runs 39221870768 and 39222297525, both concluded cancelled.
    • The next smoke, 39222454134, wrote success at 2026-10-19T08:32Z; this seat read it.
    • The dev's census of 299 workflow_run smokes (2026-10-16T00:25Z → 2026-10-19T08:23Z) counted 224 success, 75 cancelled and 0 failure, and every cancelled run was superseded by a newer one.
  • Gates, pinned to head 0c5ad5810f. 38 check-runs, all completed: 26 success, 12 skipped. node scripts/pm/check-expected-skips.mjs --pr 24503 → VERDICT check-expected-skips: OK — 12 skipped check-run(s), every one in the roster (exit 0). The dev's 39 derived local families all exit 0 (--ran reconciled; check:workflow-status-functions and check:workflow-step-name-quoting among them).
  • Route choice, accepted. (b) pending, not (a) silence and not error. The four-axis analysis is in the PR body: silence leaves a false "Installing…" pending behind, and error renders red like failure.
  • Acceptance notes, not filed. A smoke that overruns timeout-minutes is also cancelled and now reads pending, not red; the dev measured 0 such runs (the longest cancelled run took 9.1 min against a 45 min limit). A cancel with no later smoke leaves an honest pending.
  • Path fork. node scripts/pm/check-governed-merges.mjs --pr 24503 → GOVERNED, Tier H, 19 changed lines. This record, needs-user-decision on the PR, review requested from os-zhuang and hotlong. ⛔ Not readied or queued until an authorised approval.
  • Done-when 2 (the first superseded smoke on the release branch after the merge reads pending) is this seat's post-landing reading.

Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd size/s skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

3 participants