You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
German tokenizer splits gender-inclusive colon forms (Kund:innen) #14010
The German infix rules split a colon between letters, which breaks
gender-inclusive forms, standard orthography in contemporary German
business and public-sector text, before tagging or NER ever see the word:
importspacynlp=spacy.load("de_core_news_lg") # tokenizer behavior; any de pipelineprint([t.textfortinnlp("Unsere Kund:innen und Mitarbeiter:innen sind zufrieden.")])
# ['Unsere', 'Kund', ':', 'innen', 'und', 'Mitarbeiter', ':', 'innen', 'sind', 'zufrieden', '.']print([t.textfortinnlp("Wir suchen eine:n Ärzt:in für unser Team.")])
# ['Wir', 'suchen', 'eine', ':', 'n', 'Ärzt', ':', 'in', 'für', 'unser', 'Team', '.']
Expected: Kund:innen, Mitarbeiter:innen, eine:n, Ärzt:in as single
tokens. For comparison, the star and interpunct variants already stay
whole (Kolleg*innen, Expert·innen), only the colon splits, via this
infix in spacy/lang/de/punctuation.py:
r"(?<=[{a}])[:<>=](?=[{a}])".format(a=ALPHA)
The gender colon is the most common separator in German inclusive writing
(the INCLUSIFY benchmark, arXiv:2212.02564, documents frequency across sources). Every downstream component degrades on the split: the noun is tagged in pieces, NER never sees the name-like whole, and lemmas are
computed for the fragments. We created an inclusive language text checker and patch this infix; we'd like to upstream the behavior.
Two possible fixes, happy to submit a PR for either:
Conservative: exempt the noun gender endings from the colon infix - r"(?<=[{a}]):(?!in(nen)?\b)(?=[{a}])" - which keeps Kund:in and Kund:innen whole. Article forms (eine:n, jede:r) would still
split; covering them too means exempting short lowercase tails, e.g. r"(?<=[{a}]):(?![a-zäöüß]{1,5}\b)(?=[{a}])", at the cost of no longer
splitting all-lowercase typos like wort:wort.
A tokenizer config option, if changing the default is unwanted.
Your Environment
Operating System: macOS-26.6.1-arm64-arm-64bit
Python Version Used: 3.12.9
spaCy Version Used: 3.8.14
Environment Information: de_core_news_lg 3.8.0 (behavior identical on
3.7.x; long-standing default, not a regression)
Notes
The original solution was developed without AI assistance; however, this ticket was written with the help of Claude. I was unable to find an AI policy for this project.
Our production patch keeps <>= splitting and drops only the colon.
I have another ticket in the pipeline related to ADJA mistagging of colon forms, which is only observable once they survive tokenization.
Activity