Repository navigation
[Bug][GitHub] Closed (unmarked) Pull Requests are re-fetched on every run #9240
Description
Search before asking
- I had searched in the issues and found no similar issues.
What happened
One of my repo failed last night pipeline on the Collect Pull Requests task on a PR that had been closed (not merged) for more than a month and had not been updated since. For the story GitHub returned a 422 on it.
That made me take a deeper look as there was no reason to re-fetch that PR.
it looks like every closed but not merged PR is being re-fetched on every incremental run. On my instance that's about ~15k closed PRs.
For example, one of the oldest PR in my DB was closed in August 2019, has not been updated since, however at every run:
SELECT number, state, merged, github_created_at, closed_at, github_updated_at
FROM _tool_github_pull_requests
WHERE state = 'closed' AND merged = 0 AND merged_at IS NULL
ORDER BY github_updated_at ASC LIMIT 1;number,state,merged,github_created_at,closed_at,github_updated_at
29,closed,0,2019-08-02 05:39:12,2019-08-08 08:31:19,2019-08-08 08:31:19
SELECT id, created_at, url FROM _raw_github_api_pull_requests
WHERE url LIKE '%/[REPOSITORY]/pulls/29'
ORDER BY created_at DESC LIMIT 7;id,created_at,url
5014408,2026-10-19 00:27:23,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4992046,2026-10-18 00:27:07,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4969613,2026-10-17 00:27:19,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4947192,2026-10-16 00:27:38,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4924806,2026-10-15 00:27:32,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4902464,2026-10-14 00:27:21,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4880192,2026-10-13 00:27:17,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
183 identical fetches over the last 182 days of raw data we retain, for a PR untouched for 7 years.
Sanitized error:
subtask Collect Pull Requests ended unexpectedly
caused by: Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29.
The last error was: Http DoAsync error calling [method:GET path:repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 query:map[]].
Response: {"message":"The request could not be processed because too many files changed","documentation_url":"https://docs.github.com/rest/pulls/pulls#get-a-pull-request","status":"422"} (422)
attached stack trace
-- stack trace:
| github.com/apache/devlake/core/runner.RunPluginSubTasks
| /app/core/runner/run_task.go:341
| [...repeated from below...]
Wraps: (2) subtask Collect Pull Requests ended unexpectedly
Wraps: (3) attached stack trace
-- stack trace:
| github.com/apache/devlake/helpers/pluginhelper/api.(*WorkerScheduler).WaitAsync
| /app/helpers/pluginhelper/api/worker_scheduler.go:173
| github.com/apache/devlake/helpers/pluginhelper/api.(*ApiCollector).Execute
| /app/helpers/pluginhelper/api/api_collector.go:206
| github.com/apache/devlake/helpers/pluginhelper/api.(*StatefulApiCollector).Execute
| /app/helpers/pluginhelper/api/api_collector_stateful.go:97
| github.com/apache/devlake/plugins/github/tasks.CollectApiPullRequests
| /app/plugins/github/tasks/pr_collector.go:141
| github.com/apache/devlake/core/runner.runSubtask
| /app/core/runner/run_task.go:433
| github.com/apache/devlake/core/runner.RunPluginSubTasks
| /app/core/runner/run_task.go:338
| github.com/apache/devlake/core/runner.RunPluginTask
| /app/core/runner/run_task.go:173
| github.com/apache/devlake/core/runner.RunTask
| /app/core/runner/run_task.go:147
| github.com/apache/devlake/server/services.runTaskStandalone
| /app/server/services/task_runner.go:114
| github.com/apache/devlake/server/services.RunTasksStandalone.func1
| /app/server/services/task.go:187
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1771
Wraps: (4) Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29. ...
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
### What do you expect to happen
A PR should only be re-fetched individually when it has actually changed since it was last collected, else in a few years time will be fetching way more closed than opened PR.
Reopening, pushing, commenting, reviewing, merging or closing a PR all bump `updated_at`, so if a closed PR gets re-opened Devlake should pick it up.
### How to reproduce
1. Collect a repository that has closed-unmerged PRs.
2. Run incremental collection again.
3. Observe one `GET /repos/{org}/{repo}/pulls/{number}` per open **and** per closed-unmerged
PR in the database, on every run, including PRs untouched for years:
```sql
SELECT COUNT(*) FROM _tool_github_pull_requests
WHERE state != 'closed' OR merged = 0;
Anything else
Happens at every collection, and I assume collection time will increase the more unmerged closed PR there are.
Also DB size will become bloated with duplicated data seeing that each non-updated run stores the same payload in _raw_github_api_pull_requests.
Hope I didn't make obvious mistakes in that analysis.
Version
v1.0.3-beta18
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity