Skip to content

Ask one predicate per question for split credentials, numerals and paired initials (#752) - #884

Merged
derek73 merged 11 commits into
masterfrom
feat/752-phase2b-shared-predicates
Oct 25, 2026
Merged

derek73 merged 11 commits into
masterfrom
feat/752-phase2b-shared-predicates

Conversation

@derek73

@derek73 derek73 commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Part of #752: one shared predicate for each of three questions that #752's later changes widen. Neutral: no reading moves.

What changes

All three predicates are in nameparser/_pipeline/_vocab.py.

Neutrality

Measured against this branch on 632685c, and again on 2dd0414 after phase 2a (#883) merged:

  • Grid. 20,364 texts under all three orders read identically, compared on fields, ambiguity kinds and details, token roles and tags, initials() and capitalized(). The grid covers:
    • 22 heads: no comma, a whole name before a comma, one word before it, nothing before it, a nickname-only head, a maiden clause, a family comma with a title-particle head, and a quoted clause before the pair.
    • 33 spellings, among them MA, M.D., X.Y.Z., XYZ, Ph. D. and its case and spacing variants, V, VI, III, XI, M.J., K.C. and พ.บ..
    • Each text also written in lower and in upper case.
  • The harness sees differences. With each piece knocked out by monkeypatch, the same harness reported 45,871 (split_pair), 18 (C12's call to the numeral predicate), 3,974 (the cased half) and 2,986 (the peel's call to it) readings differing.
  • Direct comparison. The old and new predicates gave the same answer on 19,769 direct calls.
  • Corpora:
    • tests/v2/test_snapshot.py doesn't move.
    • Every cldr_agreement.py record matches: CLDR, GND, Wikidata, conventions, native, and both locale views.
    • local_agreement.py psc-individuals matches its record. The record was already stale against 632685c (13 records moved before this change), so I re-recorded it from each base first.
    • The local snapshots of ctgov-officials, fec-individuals and tse-donors, each recorded from the base with --tree, all match on both bases.
  • Frames (sys.setprofile):
    • Unchanged: _CALL_BASELINE's reference name (305) and Smith, John (158).
    • +1 per site that tests the split pair, for each Ph word with a word after it: John Smith Ph. D. goes 233 → 234, Smith, John Ph. D. 239 → 241.
    • Unchanged on C12's evidence: John Smith Jr., AD stays at 372, since the tag test runs ahead of the call.
    • The split test's frame is paid whether or not a D. follows (ph ph ph ph ph ph 282 → 286), so it grows only with such words. It's a rare shape that no guard counts. uv run pytest -n0 -m serial passed three runs in a row. It failed once under a load average of about 10 from a parallel session, a different clock-timed row each time.

Docs

  • mechanisms.md#ONE-PREDICATE-PER-QUESTION names the three predicates.
  • decisions/vocabulary-model has a dated entry for the change and its measurement. It points at items 6 and 8 of the agreed-design entry from Record vocabulary phase 2's agreed design in the RFC and decisions (#752) #881 for what the later changes widen. It also leaves one overlap for the split-credential PR to state: that entry plans for the split merge to ask is_cased_paired_initials, but its own item 6 refuses a split credential that opens with a single letter, so the length test answers first for A. B..
  • Six decisions entries and AGENTS.md's spelling sweep now point at the new name of the numeral predicate. decisions/vocabulary-exclusions and a cases.py note no longer name the retired PH/D patterns.
  • No stage's Reads: line changes, because no stage reads a new field.

Reviews

  • Code review. Its own grid of 71,382 texts gave 1,070,730 parses across the default policy, lenient_comma_suffixes=False, unlisted_dotted_suffixes=False, CapsSuffixes.EVERYWHERE/OFF, a delimiter policy and all three orders. It found 0 differences and no missed copy.
  • Design-docs review, then a second review of its fix commits. Between them they found the catalog overstating the numeral predicate, the sites it misses, and stale PH/D names. Each is fixed in its own commit.
  • PR review (comments, tests, simplification). It tightened comments. It also found that no test held a split credential without its periods: making either period required passed the whole suite. This PR adds four parity rows (Ph D, Ph. D, Smith, John Ph D, John Smith, PH D, each checked against the 1.4.0 wheel) and unit tests for SPLIT_PAIR_OPENS and is_cased_paired_initials. It moved C12's suffix-word test back beside the call, as above.

Checks

  • uv run pytest
  • uv run pytest -n0 -m serial
  • uv run --python 3.15 pytest -q
  • uv run mypy
  • uv run ruff check
  • sphinx-build -b doctest docs
  • python -m doctest README.rst

Part of #752; doesn't close it.

🤖 Generated with Claude Code

derek73 and others added 6 commits October 25, 2026 00:53
…ired initials (#752)

Neutral. `_vocab.split_pair` replaces the three copies of the Ph./D.
test (group's merge, `_comma._pieces`, `is_wholly_suffix`), each asking
`SPLIT_PAIR_OPENS` C-level first so an ordinary word pays no frame.
`_vocab.reads_as_generation` replaces `is_trailing_numeral_suffix` and
takes the vocabulary path as `listed`, which C12's evidence passes.
`_vocab.is_cased_paired_initials` lifts #872's inline script condition
out of classify; C5 keeps the bare `is_paired_initials`.

The regex roster named `_group._PH` and `_group._D` after both had moved
to `_vocab`; a new test refuses an entry naming no pattern.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mechanisms.md's ONE-PREDICATE-PER-QUESTION names the three predicates,
decisions/vocabulary-model records the change, its neutrality
measurement and what the later #752 changes widen, and the decisions
entries and AGENTS.md naming the old numeral predicate point at its new
name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

The design-docs review found the catalog calling `reads_as_generation`
the numeral's one recognizer while S8's run start, C1's listing form and
the peel's vocabulary path still recognize a numeral on their own; that
`listed` carries C12's whole suffix-word tag, not a numeral test; that
the peel's `is_suffix_piece` vetoes `i` and `v`, so passing `listed` at
its fork would move `John J. V`; that the split-credential widening
never reaches `is_cased_paired_initials` for `A. B.`; and two notes
still naming the retired `PH`/`D`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The code review counted one frame per Ph word with a word after it,
whether a D. follows or not; the entry had said per standing pair.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second review found the list of numeral sites missing C9's
family-comma walk and the one-letter numeral behind a suffix word there,
and the comment above reads_as_generation still calling it the
numeral's one recognizer. The list is now stated as found by search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ared-predicates

# Conflicts:
#	docs/design/decisions/vocabulary-model.md
@derek73 derek73 added tests design vocabulary Per-word vocabulary model and language packs labels Oct 25, 2026
@derek73 derek73 self-assigned this Oct 25, 2026
@codecov

codecov Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.08%. Comparing base (2dd0414) to head (db694cb).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #884   +/-   ##
=======================================
  Coverage   99.07%   99.08%           
=======================================
  Files          47       47           
  Lines        4781     4785    +4     
=======================================
+ Hits         4737     4741    +4     
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@derek73 derek73 added this to the 2.7 milestone Oct 25, 2026
derek73 and others added 5 commits October 25, 2026 01:53
…ared-predicates

# Conflicts:
#	docs/design/decisions/vocabulary-model.md
Merging master put two 2026-10-25 #752 log entries in
decisions/vocabulary-model, so the Standing line pointing at 2a's
names it by its opening words.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The peel's veto is on the initial tag, not the shape; the numeral
predicate's summary says what listed makes it answer; the split pre-test
holds only while the first-word test is _PH; group alone declines a head
pair; the site list lives in decisions/ only; import order and a
cases.py note naming _vocab.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…plit (#752)

`reads_as_generation` no longer takes C12's suffix-word tag as `listed`:
C12 asks whether the word ending the part is any suffix word or reads
as a generation, and `Jr.` and `Esq.` are no numerals, so the tag test
stands beside the call again, as on master, and costs master's frames.
The listed numerals belong in the predicate, but reading them from the
lexicon now would move `John J. V`, so that half waits for the change
that has every numeral rule ask it; the decisions entry says so.

The test review found no row holding a split credential without its
periods, so making either period required passed the suite: four parity
rows (`Ph D`, `Ph. D`, `Smith, John Ph D`, `John Smith, PH D`), a
periodless pair in `test_is_wholly_suffix`, and unit tests that
`SPLIT_PAIR_OPENS` is a necessary condition of `split_pair` and that
`is_cased_paired_initials` refuses Thai `พ.บ.`.

Re-measured on 2dd0414: the grid, the committed snapshot, every
agreement record, PSC and the ctgov, FEC and tse local snapshots match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hem moves (#752)

The review of the fix commits found that the shape test already takes
ii to iv; only the one-letter i and v would be new, and they move 375
readings at the peel's fork and 144 reports at the numeral-before-a-
credential report, not the peel alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit 2af8da8 into master Oct 25, 2026
12 checks passed
@derek73
derek73 deleted the feat/752-phase2b-shared-predicates branch October 25, 2026 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

design tests vocabulary Per-word vocabulary model and language packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant