Skip to content

Split local_corpora.py into a shared core and one module per source - #855

Merged
derek73 merged 3 commits into
masterfrom
refactor/split-local-corpora
Oct 23, 2026
Merged

derek73 merged 3 commits into
masterfrom
refactor/split-local-corpora

Conversation

@derek73

@derek73 derek73 commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Follows #854, which deferred this split. tools/differential/local_corpora.py (2,374 lines) becomes:

  • local_core.py: what the sources share (Fetched, Source, the retry policy, the share gate, _download, fetch_command, and an unchecked directory)
  • one module per source: local_ctgov.py, local_fec.py, local_tse.py, local_tw.py, local_ukec.py, local_psc.py, each opening with the description of its source that used to be in local_corpora's docstring
  • local_corpora.py (about 820 lines): the SOURCES registry, the cache's files (fetch, load, read_records), the views, the word index, the fingerprints and the command line

Every documented tools/differential/local_corpora.py … command and every load_tool("local_corpora") works unchanged, and so do the imports in name_usage.py, snapshot.py and local_agreement.py. No behavior moves.

Commits, for blame (merge without squashing)

  1. The move only. Line ranges are cut verbatim into the new files; the only new lines are each module's docstring head and imports. git blame -C -C on a moved line reaches the commit that wrote it (for example _backed_off → Simplify the local corpora: one retry policy, one share gate, shared record and fingerprint helpers #854, the fec-individuals paragraph → Add the FEC's individual contributors as a local corpus #834). This commit does not pass the suite on its own, because the tests still patch the old module. Its message says so.
  2. The edits. Repointing the test patches, qualified reads in local_tse, a guard test, and docs.
  3. From the review. Each constant is bound in exactly one module, the guard checks every module pair, and moved comments that named something now in another module say which.

A squash merge would fold the move into the edits and lose the blame trail, so this needs a merge commit or a rebase merge. If a later git bisect lands on d666bb9, it needs git bisect skip.

Monkeypatches

A patch only reaches a fetch when it is set on the module whose globals the fetch reads. So the tests' 43 patch sites on a moved constant now target that module: local_core for RETRIES and CHUNK, local_fec for FEC_MAX_BAD, and so on. The test module takes these modules from sys.modules, because a second load by path would be a copy that no fetch reads. local_tse reads local_core.RETRIES, USER_AGENT and _TRANSIENT through the module, as _backed_off does, so one patch reaches every reader.

No module from-imports an upper-case name from a sibling module, except PSC into local_corpora. local_corpora's credit lines read FEC_CYCLE and TSE_ELECTIONS through their modules. As a result, a stale setattr(local_corpora, "TSE_MAX_BAD", …) raises instead of passing with its guard off. test_no_module_imports_a_sibling_constant holds this over the modules SOURCES names, so a new source is covered. Its negative controls fail it: importing USER_AGENT into local_tse again, or FEC_CYCLE into local_corpora.

Verification

  • Negative controls retaken. Each of the 43 repointed patch sites was removed one at a time and the test module run, on master (48ed448) and on this branch. All 43 sites give the same failing tests on both trees: 35 fail tests when removed, and 8 made no test fail before the split either (those are pre-existing, not touched here).
  • Fingerprints unchanged. fold_fingerprint() and local_agreement.scorer_fingerprint() are byte-identical to master, so cached word indexes and the PSC agreement record stay valid.
  • Same answers on all six cached corpora (hashes only, no names printed): records, meta, header(), tally, cached and freshly built word index, lookups, pair_counts for tse-donors, and name_usage.py's local rows are identical to master. So is the output of local_corpora.py fetch <source> (cached path) and pairs tse-donors.
  • Read back through the fetch functions. fetch_fec() and fetch_psc(), run on their cached inputs with the network refused, give records whose sha256, detail and data date match meta.json. tw, ukec, tse and ctgov fetch from the network, so they were checked only through the comparisons above.
  • local_agreement.py psc-individuals matches its record.
  • ruff, mypy and the full suite pass on 3.11 (23,095 passed). The local-corpus, agreement, name_usage and snapshot test modules pass on 3.15.

One pre-existing finding, not from this branch: snapshot.py --local ukec-individuals reports 508 moves on master as well, with identical output. The cached local snapshot predates recent title changes and needs re-recording.

🤖 Generated with Claude Code

derek73 and others added 2 commits October 23, 2026 13:55
A move only, kept apart from the edits so that `git blame -C` follows
every moved line back to the commit that wrote it: local_corpora.py's
shared fetch machinery goes to local_core.py, and each source's
constants, functions and docstring paragraph to local_ctgov.py,
local_fec.py, local_tse.py, local_tw.py, local_ukec.py and local_psc.py.
local_corpora.py keeps the SOURCES registry, the cache's own files, the
views and the command line, importing the sources to build SOURCES. The
new lines are the modules' docstring heads and imports, and local_core's
unchecked directory().

The tests still patch the moved constants on local_corpora, which no
longer reads them, so this commit does not pass the suite on its own:
the next one repoints the patches and the one constant a source reads
from local_core.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edits after the move. A patch reaches a fetch only when set on the
module whose globals the fetch reads, so each test's patch of a source
constant moves from local_corpora to that source's module (local_fec,
local_tse, ...) and RETRIES and CHUNK to local_core, and the test module
takes those modules from sys.modules: a second load by path would be a
copy no fetch reads. local_tse reads local_core.RETRIES through the
module, as _backed_off does, so one patch still sets both.

local_corpora binds no source's constant: the credit lines read
FEC_CYCLE and TSE_ELECTIONS through their modules, so a stale patch on
local_corpora raises instead of passing with its guard off, and
test_local_corpora_binds_no_constant_a_fetch_reads holds that.
local_corpora.directory, the checked one, delegates to local_core's.
The docstring, tools/differential/README.md and AGENTS.md say where a
source's code and description now live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.05%. Comparing base (48ed448) to head (1fe3fad).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #855   +/-   ##
=======================================
  Coverage   99.05%   99.05%           
=======================================
  Files          47       47           
  Lines        4669     4669           
=======================================
  Hits         4625     4625           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…at its new home

From the review of the split. local_tse read USER_AGENT and _TRANSIENT
through a `from local_core import`, so a patch on local_core would have
missed TSE's range reader; it reads them through the module now, as it
reads RETRIES. The guard test asks the property directly, over the
modules SOURCES names: no module from-imports an upper-case name from a
sibling, but PSC into local_corpora. Removing either fix fails it.

Moved comments that named something now in another module say which
(local_corpora.load, local_fec.FEC_MAX_BAD, local_corpora.Local.records,
local_core's retry policy), and the rosters of jsonl_line's users in
corpus_build.py and AGENTS.md name the modules that use it. The dead
TYPE_CHECKING block a ruff fix left in local_corpora.py is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit 1d118c9 into master Oct 23, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant