Repository navigation
Split local_corpora.py into a shared core and one module per source - #855
Merged
Merged
Conversation
A move only, kept apart from the edits so that `git blame -C` follows every moved line back to the commit that wrote it: local_corpora.py's shared fetch machinery goes to local_core.py, and each source's constants, functions and docstring paragraph to local_ctgov.py, local_fec.py, local_tse.py, local_tw.py, local_ukec.py and local_psc.py. local_corpora.py keeps the SOURCES registry, the cache's own files, the views and the command line, importing the sources to build SOURCES. The new lines are the modules' docstring heads and imports, and local_core's unchecked directory(). The tests still patch the moved constants on local_corpora, which no longer reads them, so this commit does not pass the suite on its own: the next one repoints the patches and the one constant a source reads from local_core. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edits after the move. A patch reaches a fetch only when set on the module whose globals the fetch reads, so each test's patch of a source constant moves from local_corpora to that source's module (local_fec, local_tse, ...) and RETRIES and CHUNK to local_core, and the test module takes those modules from sys.modules: a second load by path would be a copy no fetch reads. local_tse reads local_core.RETRIES through the module, as _backed_off does, so one patch still sets both. local_corpora binds no source's constant: the credit lines read FEC_CYCLE and TSE_ELECTIONS through their modules, so a stale patch on local_corpora raises instead of passing with its guard off, and test_local_corpora_binds_no_constant_a_fetch_reads holds that. local_corpora.directory, the checked one, delegates to local_core's. The docstring, tools/differential/README.md and AGENTS.md say where a source's code and description now live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #855 +/- ##
=======================================
Coverage 99.05% 99.05%
=======================================
Files 47 47
Lines 4669 4669
=======================================
Hits 4625 4625
Misses 44 44 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…at its new home From the review of the split. local_tse read USER_AGENT and _TRANSIENT through a `from local_core import`, so a patch on local_core would have missed TSE's range reader; it reads them through the module now, as it reads RETRIES. The guard test asks the property directly, over the modules SOURCES names: no module from-imports an upper-case name from a sibling, but PSC into local_corpora. Removing either fix fails it. Moved comments that named something now in another module say which (local_corpora.load, local_fec.FEC_MAX_BAD, local_corpora.Local.records, local_core's retry policy), and the rosters of jsonl_line's users in corpus_build.py and AGENTS.md name the modules that use it. The dead TYPE_CHECKING block a ruff fix left in local_corpora.py is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #854, which deferred this split.
tools/differential/local_corpora.py(2,374 lines) becomes:local_core.py: what the sources share (Fetched,Source, the retry policy, the share gate,_download,fetch_command, and an uncheckeddirectory)local_ctgov.py,local_fec.py,local_tse.py,local_tw.py,local_ukec.py,local_psc.py, each opening with the description of its source that used to be inlocal_corpora's docstringlocal_corpora.py(about 820 lines): theSOURCESregistry, the cache's files (fetch,load,read_records), the views, the word index, the fingerprints and the command lineEvery documented
tools/differential/local_corpora.py …command and everyload_tool("local_corpora")works unchanged, and so do the imports inname_usage.py,snapshot.pyandlocal_agreement.py. No behavior moves.Commits, for blame (merge without squashing)
git blame -C -Con a moved line reaches the commit that wrote it (for example_backed_off→ Simplify the local corpora: one retry policy, one share gate, shared record and fingerprint helpers #854, the fec-individuals paragraph → Add the FEC's individual contributors as a local corpus #834). This commit does not pass the suite on its own, because the tests still patch the old module. Its message says so.local_tse, a guard test, and docs.A squash merge would fold the move into the edits and lose the blame trail, so this needs a merge commit or a rebase merge. If a later
git bisectlands on d666bb9, it needsgit bisect skip.Monkeypatches
A patch only reaches a fetch when it is set on the module whose globals the fetch reads. So the tests' 43 patch sites on a moved constant now target that module:
local_coreforRETRIESandCHUNK,local_fecforFEC_MAX_BAD, and so on. The test module takes these modules fromsys.modules, because a second load by path would be a copy that no fetch reads.local_tsereadslocal_core.RETRIES,USER_AGENTand_TRANSIENTthrough the module, as_backed_offdoes, so one patch reaches every reader.No module
from-imports an upper-case name from a sibling module, exceptPSCintolocal_corpora.local_corpora's credit lines readFEC_CYCLEandTSE_ELECTIONSthrough their modules. As a result, a stalesetattr(local_corpora, "TSE_MAX_BAD", …)raises instead of passing with its guard off.test_no_module_imports_a_sibling_constantholds this over the modulesSOURCESnames, so a new source is covered. Its negative controls fail it: importingUSER_AGENTintolocal_tseagain, orFEC_CYCLEintolocal_corpora.Verification
fold_fingerprint()andlocal_agreement.scorer_fingerprint()are byte-identical to master, so cached word indexes and the PSC agreement record stay valid.header(),tally, cached and freshly built word index, lookups,pair_countsfor tse-donors, andname_usage.py's local rows are identical to master. So is the output oflocal_corpora.py fetch <source>(cached path) andpairs tse-donors.fetch_fec()andfetch_psc(), run on their cached inputs with the network refused, give records whose sha256, detail and data date matchmeta.json. tw, ukec, tse and ctgov fetch from the network, so they were checked only through the comparisons above.local_agreement.py psc-individualsmatches its record.ruff,mypyand the full suite pass on 3.11 (23,095 passed). The local-corpus, agreement, name_usage and snapshot test modules pass on 3.15.One pre-existing finding, not from this branch:
snapshot.py --local ukec-individualsreports 508 moves on master as well, with identical output. The cached local snapshot predates recent title changes and needs re-recording.🤖 Generated with Claude Code