Skip to content

Add retries and timeout to Grafana request in flaky test detection - #40669

Open
schizophrenicmaniac wants to merge 1 commit into
apache:masterfrom
schizophrenicmaniac:fix/40662-the-flaky-test-detection-job-is-flaky
Open

schizophrenicmaniac wants to merge 1 commit into
apache:masterfrom
schizophrenicmaniac:fix/40662-the-flaky-test-detection-job-is-flaky

Conversation

@schizophrenicmaniac

Copy link
Copy Markdown
Contributor

The flaky test detection job calls the Grafana alertmanager API with a plain requests.get. That call has no timeout and no retry, so any blip on metrics.beam.apache.org fails the whole run. This change uses a session with urllib3 Retry: 5 retries with backoff on connection errors and 429/5xx responses, plus a 30s timeout. If Grafana is still failing after that, the run fails as it did before.

get_workflow_issues() also called issue.get_labels() for every open and closed flaky_test issue, which is one extra API request per issue. The labels already come with the issue listing, so it now reads issue.labels.

I added test_flaky_test_detection.py next to the existing tool tests. It covers alert parsing, retrying after a 503, no retry on a 404, the timeout, and the label lookup. The Grafana tests run against a small local HTTP server, so the real retry path is exercised.

Fixes #40662

@github-actions github-actions Bot added the infra label Oct 19, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR fixes a TypeError in flaky_test_detection.py caused by GitHub returning None for the body of empty issues, and optimizes label checks by inspecting the already-loaded issue.labels property instead of issuing extra API requests via issue.get_labels(). The fix is clean, directly addresses the failure, and includes comprehensive unit test coverage for the affected issue-parsing logic.

🔍 General Feedback

  • The null-safe fallback (issue.body or "") and the switch from issue.get_labels() to issue.labels are idiomatic and well-tested in test_flaky_test_detection.py.
  • No blocking issues or regressions were found.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @Abacn added as fallback since no labels match configuration
Backup reviewer: chamikaramj (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The Flaky test detection job is flaky

1 participant