Skip to content

ci: reduce SDK CI critical paths and runner saturation - #2480

Merged
SteveSandersonMS merged 5 commits into
mainfrom
copilot/reduce-ci-runner-saturation
Sep 2, 2026
Merged

SteveSandersonMS merged 5 commits into
mainfrom
copilot/reduce-ci-runner-saturation

Conversation

@SteveSandersonMS

@SteveSandersonMS SteveSandersonMS commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • cancel superseded SDK workflow runs when a pull request is updated
  • stop rerunning each language SDK matrix on main after the same tree passed merge-queue validation, avoiding about 293 runner-minutes per merge
  • shorten the normal full-run critical path by optimizing .NET matrix builds and shard balance, removing duplicate Java work, parallelizing Rust quality jobs, and eliminating a deterministic Go replay-proxy delay
  • bound all Go, Python, Rust, and .NET jobs at 20 minutes so an isolated hang cannot occupy a runner for 35–50 minutes

Investigation

Recent full merge-group runs normally complete in about 11:35–13:07. The 35–65 minute outliers combined two problems: individual legs waited as long as 54 minutes for runners, and occasional test legs hung for 35–48 minutes. A full run creates roughly 48 substantive jobs and consumes 280–300 runner-minutes, so redundant and superseded matrices amplify queue pressure.

Measured normal critical-path legs were .NET Windows shard 2 (~12:01), Go Windows default (~10:19), Rust Ubuntu default (~10:10), Java Windows ARM64 in-process (~9:32), and Python Windows in-process (~9:16).

Changes and expected impact

  • .NET: run full format/analyzer/build validation once in parallel, build only test-consumed frameworks in matrix cells, and rebalance the slow shard. Expected critical-path reduction: about 2–3 minutes.
  • Java: avoid four known 30-second subprocess-oriented timeouts in every in-process classifier cell, skip successful report generation, and avoid rebuilding the native reactor in JDK 25 after classifier coverage. Expected critical-path reduction: about 3 minutes.
  • Rust: run Linux-only format/Clippy and docs as parallel jobs instead of serial phases in the Ubuntu test cell. Expected critical-path reduction: about 2–3 minutes.
  • Go: initialize the replay proxy before the unknown-session test, removing a deterministic ~76-second retry delay from every matrix cell.

Because these jobs run concurrently, the reductions are not additive. The expected normal all-language wall time is roughly 9–10 minutes instead of about 12 minutes, while the concurrency and post-merge changes reduce queue pressure much more substantially.

Coverage and validation

  • Full SDK coverage remains for pull requests, merge groups, and manual dispatches; this does not add path filtering.
  • Merge-group checks remain attached to the exact commit that lands on main, so normal queued commits still show their green/red SDK status without a redundant rerun.
  • Direct pushes to main no longer start standalone SDK matrices.
  • .NET format and full build passed; both platform shard partitions cover all 756 discovered tests exactly once, and the optimized shard command passed 421 tests with 2 skips.
  • Workflow formatting and diff-integrity checks passed.
  • Java execution validation is intentionally left to this draft PR CI run after the local Maven validation proved unusually slow.

SteveSandersonMS and others added 2 commits September 2, 2026 09:17
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SteveSandersonMS SteveSandersonMS changed the title ci: reduce SDK runner saturation ci: reduce SDK CI critical paths and runner saturation Sep 2, 2026
Comment thread .github/workflows/rust-sdk-tests.yml Fixed
Comment thread .github/workflows/rust-sdk-tests.yml Fixed
Comment thread .github/workflows/rust-sdk-tests.yml Fixed
Comment thread .github/workflows/rust-sdk-tests.yml Fixed
Comment thread .github/workflows/rust-sdk-tests.yml Fixed
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SteveSandersonMS
SteveSandersonMS marked this pull request as ready for review September 2, 2026 09:53
@SteveSandersonMS
SteveSandersonMS requested a review from a team as a code owner September 2, 2026 09:53
Copilot AI balanced review requested due to automatic review settings September 2, 2026 09:53
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Restore Java site-artifact publication and address or document the consistency-review model pin.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: 2 Medium severity

New issues introduced by this change (2)
Severity Finding
Medium severity .github/​workflows/​java-sdk-tests.yml — Removing the push trigger makes the Upload test results for site generation step at lines…
Medium severity .github/​workflows/​sdk-consistency-review.md — This unrelated frontmatter change hard-codes both the review model and the threat-detection model…
What changed in this PR

Reduces SDK CI latency and runner saturation while preserving merge-queue validation.

Changes:

  • Cancels superseded runs, removes redundant main matrices, and adds timeouts.
  • Optimizes .NET, Java, Rust, and Go execution.
  • Updates the SDK consistency reviewer model.
File Review
java/​sdk/​pom.xml Excludes duplicated subprocess tests from in-process cells.
go/​internal/​e2e/​rpc_server_e2e_test.go Initializes the replay proxy earlier.
.github/​workflows/​sdk-consistency-review.md Model pin bypasses repository-level overrides; remove or document it.
.github/​workflows/​sdk-consistency-review.lock.yml Regenerates the compiled workflow.
.github/​workflows/​rust-sdk-tests.yml Parallelizes quality jobs and adds timeouts.
.github/​workflows/​required-checks.yml Cancels superseded PR runs.
.github/​workflows/​python-sdk-tests.yml Removes main runs and adds a timeout.
.github/​workflows/​nodejs-sdk-tests.yml Removes redundant main runs.
.github/​workflows/​java-sdk-tests.yml Site test-result publication becomes unreachable in normal automation.
.github/​workflows/​go-sdk-tests.yml Removes main runs and adds a timeout.
.github/​workflows/​dotnet-sdk-tests.yml Optimizes validation and shards; the format failure branch cannot emit its intended diagnostic.
Suppressed comments (1)

.github/workflows/dotnet-sdk-tests.yml:38

  • GitHub Actions invokes shell: bash with -e, so when dotnet format fails the step exits immediately and never reaches this custom diagnostic branch. Wrap the command directly in the if condition so contributors receive the intended remediation message.
          dotnet format --no-restore --verify-no-changes
          if [ $? -ne 0 ]; then
            echo "❌ dotnet format produced changes. Please run 'dotnet format' in dotnet"
            exit 1
          fi

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/java-sdk-tests.yml
Comment thread .github/workflows/sdk-consistency-review.md Outdated
@github-actions github-actions Bot mentioned this pull request Sep 2, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SteveSandersonMS
SteveSandersonMS merged commit 975cf6b into main Sep 2, 2026
73 of 74 checks passed
@SteveSandersonMS
SteveSandersonMS deleted the copilot/reduce-ci-runner-saturation branch September 2, 2026 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants