Split local_corpora.py into a shared core and one module per source

AGENTS.md

@@ -311,9 +311,9 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_
 
 **Perf regressions are caught by the scaling test, not the absolute-time ones** — `tests/v2/test_benchmark.py::test_parse_cost_grows_no_worse_than_linearly` times a repeated unit at n vs 4n over a table of shapes (one per pipeline inner loop) and bounds the ratio; the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. Two rules when touching it: calibrate `_MAX_RATIO` against the WEAKEST quadratic's signal (a mixed quadratic surfaces far below the textbook 16×, so the operating point `_BASE` matters more than the bound), and confirm a planted regression fails it across REPEATED runs — one failure is a coin-flip on a timing test. The shapes cover different dimensions (segment count only via `commas`, intra-piece accumulation only via `particles`/`conjunctions`, non-ASCII input only via `honorifics` — every other unit is pure ASCII, so `script_segment` returns at its bail and the CJK stages go unmeasured, M2's clause view only via `maiden_clause`, the one unit holding a marker (until #601 its order mattered, `MA nee ` reaching nothing; since #601 either order reaches the take) — and connective RUN LENGTH only via `link_run`, whose unit has to be MIXED CASE and hold a connective of the generational class: `i und ` reads the letter as an initial and reaches nothing where `i Und ` reaches everything, and S7's anchor pass only via `credential_run`, whose unit has to hold a Title-case member behind an unambiguous credential: `PhD MA ` reads the same fields and never asks the pass, the capitals deciding each member first); measure before pruning one. **A shape whose input needs a PREFIX cannot be a `_SHAPES` row at all**, since that table repeats a unit and nothing else — a maiden clause needs a name word and a marker before the run it is about, and `"nee i Und " * n` reaches the clause rule not at all, measuring the identical ratio on a broken tree and a fixed one. That one is `test_a_clause_link_run_does_not_cost_quadratically`, which builds its own input and counts FRAMES. A prefixed shape whose cost is Python-level joins `_REREAD_SHAPES` beside it — (input builder, the one function the defect re-runs, reachability probe) rows counted in that function's frames at 8 units against 32 — which holds #558's trailing fixed point, its maiden-clause twin, and #559's particle chain, a shape that grows at both ends and so has no unit to repeat. A prefixed shape whose cost is C-level work — a list scanned by `in`, a tuple re-hashed as a dict key — emits no frame for that count to see, so it goes in `_PREFIXED_SHAPES` instead: (prefix, unit, reachability probe) rows timed on the clock like `_SHAPES`, at a base of their own recorded beside the table (#553, whose run shape read 5.97× once at base 800 on the pre-fix tree, under the bound; py3.11, 2026-09-28). **That table skips under a line tracer** (`sys.settrace`, coverage's core below py3.14), which slows every Python line and not the C-level scan, diluting the quadratic below the bound (5.46–5.52× broken at base 1600 under `coverage run` on py3.11, same date). The `sys.monitoring` core coverage uses from 3.14 does not dilute it (8.87–10.91× broken), and CI collects coverage in its 3.14 build job alone, so every CI job runs the table; `ja-extra` also sets `NAMEPARSER_REQUIRE_CLOCK_GUARDS` so that a line tracer there fails the rows instead of skipping them. **Coverage is collected on ONE interpreter, the one that uploads it** (`COVERAGE_ON` in `.github/workflows/python-package.yml`): the package has no version-conditional code, so one report stands for all of them, and under a line tracer the property grids in `tests/v2/test_properties.py` had cost the 3.12 job most of its ~12 minutes (measured 2026-09-29: the three #397 grids take 139s under coverage's settrace core on 3.12, 24s under its sys.monitoring core, 17s with none). **A shape the CLOCK cannot reach needs a FRAME-count guard instead**, which is the second scaling test in that file (`test_a_trailing_credential_run_does_not_cost_exponentially`, #531): where the defect is an exponential rather than a quadratic, the input length that separates the curves on a timing test does not finish, so the guard counts frames over 8 units against 16 and bounds THAT ratio. One pair does not see every curve, and the fix round for #531 measured why: at 2× the input the per-member LINEAR work swamps a quadratic (2.08× for a genuine one against 1.73× clean), so that pair guards the exponential alone and a second, longer pair — 16 against 64, where the same quadratic reads 7.42× against 3.53× clean — is what can see one. Assert them in that order: an exponential never returns from the longer run, so the cheap pair has to have failed first. **A frame guard counts ONE function, so it goes blind when the code it watches stops calling that function.** #563's paired-initials guard counted `_normalize`, which segment's scan reached; #613 moved the scan into `_comma.decide`, where it calls `is_suffix_piece`, and the quadratic came back while the guard read 3.6× and passed (review caught it: 120 → 1,620 calls for 4× the input). When code under a frame guard moves, re-take the guard's negative control on the moved code before trusting a pass. Frame counts do not move under load, so this shape needs no repeated-run calibration — but it does need the same reachability assertion `_POLICY_SHAPES` rows carry, since the walk under measurement runs only while every unit still reads as a credential. A stage gated on an opt-in `Policy` field needs a `_POLICY_SHAPES` entry instead, since bare `parse()` never enters it — and that table's rows carry a **reachability probe** run before the measurement, because a precedence change can quietly stop the shape reaching the stage and leave a green test measuring a no-op (`_POLICY_SHAPES` is also asserted non-empty: an empty `parametrize` is a skip, not a failure, so deleting its last row would retire the guard silently).
 
-**A JSONL file written with `json.dumps(..., ensure_ascii=False)` is cut by `str.splitlines()`** — `json.dumps` leaves U+2028, U+2029 and U+0085 unescaped, and `splitlines()` splits at all three, so one name pasted from a document holding a line separator turns its line into two halves of invalid JSON, and the file into one no reader can load. #801's review found it in the local corpora (a single official among 536,981 would have made the cache unloadable for good), and the review of that fix found it again one file over, in `snapshot.py`'s `dump`/`load`, which the fix had not touched. So it is fixed at the shared source now: `corpus_build.jsonl_line` escapes the three and `corpus_build.jsonl_lines` cuts at `"\n"` only, and `corpus_build`'s corpus and fields writers, `snapshot.py`, `cldr_agreement.py` and `local_corpora.py` go through them. A new JSONL writer or reader in tools/ goes through the same two. Still outside them: the builders that write their own lines (`build_cjk_corpus.py`, `_conventions`, `_shapes`, `_rules`, `_issues`), `compare.py`'s worker stream, and the tests' corpus reads (`_differential_fixtures.py`, `test_corpora.py`); none of the 16 committed JSONL files held one on 2026-10-21.
+**A JSONL file written with `json.dumps(..., ensure_ascii=False)` is cut by `str.splitlines()`** — `json.dumps` leaves U+2028, U+2029 and U+0085 unescaped, and `splitlines()` splits at all three, so one name pasted from a document holding a line separator turns its line into two halves of invalid JSON, and the file into one no reader can load. #801's review found it in the local corpora (a single official among 536,981 would have made the cache unloadable for good), and the review of that fix found it again one file over, in `snapshot.py`'s `dump`/`load`, which the fix had not touched. So it is fixed at the shared source now: `corpus_build.jsonl_line` escapes the three and `corpus_build.jsonl_lines` cuts at `"\n"` only, and `corpus_build`'s corpus and fields writers, `snapshot.py`, `cldr_agreement.py`, `local_corpora.py`, `local_psc.py`'s extracts and `local_agreement.py` go through them. A new JSONL writer or reader in tools/ goes through the same two. Still outside them: the builders that write their own lines (`build_cjk_corpus.py`, `_conventions`, `_shapes`, `_rules`, `_issues`), `compare.py`'s worker stream, and the tests' corpus reads (`_differential_fixtures.py`, `test_corpora.py`); none of the 16 committed JSONL files held one on 2026-10-21.
 
-**A row count does not prove a CSV whole: Python's `csv` returns a body cut inside a row as a short last row, without raising** -- even inside an open quote (`'a,b\r\n"x,y'` gives `[['a', 'b'], ['x,y']]`). So a fetch that checks its row count against the source's stated total, and tolerates a few malformed rows, passes a cut body one name short. #842's review found it in `ukec-individuals` (two reviewers independently); the fix requires the body to end with a line break and its last row not to be short (a cut only shortens a row; a long last row is left to the malformed share), with a test for each arm, since a quoted field holding a line break lets a cut end at one. A new CSV source in `tools/differential/local_corpora.py` takes the same check, and its skip shares are tested at their shipped values, not only patched open for a small fixture.
+**A row count does not prove a CSV whole: Python's `csv` returns a body cut inside a row as a short last row, without raising** -- even inside an open quote (`'a,b\r\n"x,y'` gives `[['a', 'b'], ['x,y']]`). So a fetch that checks its row count against the source's stated total, and tolerates a few malformed rows, passes a cut body one name short. #842's review found it in `ukec-individuals` (two reviewers independently); the fix requires the body to end with a line break and its last row not to be short (a cut only shortens a row; a long last row is left to the malformed share), with a test for each arm, since a quoted field holding a line break lets a cut end at one. A new CSV source (a `tools/differential/local_<source>.py` module) takes the same check (`local_ukec._ukec_rows`'s last-row test), and its skip shares are tested at their shipped values, not only patched open for a small fixture.
 
 **Expected-failure tests use `@pytest.mark.xfail`** — the conftest parametrized fixture breaks `@unittest.expectedFailure`; always use `@pytest.mark.xfail` instead.
 

tests/v2/_differential_fixtures.py

@@ -52,7 +52,10 @@ def load_tool(stem: str) -> ModuleType:
     --doctest-modules over the corpus builders, so these are imported
     this way rather than made importable. Each main() is behind a
     __name__ guard; the one import-time effect is that a corpus builder
-    appends its own directory to sys.path to find corpus_build.py."""
+    appends its own directory to sys.path to find corpus_build.py (and
+    local_corpora.py its local_core and local_<source> modules, which it
+    imports under their bare names: a test reaches those with
+    importlib.import_module, never a second load by path)."""
     spec = importlib.util.spec_from_file_location(
         f"differential_{stem}", _TOOLS / f"{stem}.py")
     assert spec is not None and spec.loader is not None, stem