Skip to content

German tokenizer splits gender-inclusive colon forms (Kund:innen) #14010

Open

Description

@lsmith77

How to reproduce the behaviour

The German infix rules split a colon between letters, which breaks
gender-inclusive forms, standard orthography in contemporary German
business and public-sector text, before tagging or NER ever see the word:

import spacy

nlp = spacy.load("de_core_news_lg")  # tokenizer behavior; any de pipeline
print([t.text for t in nlp("Unsere Kund:innen und Mitarbeiter:innen sind zufrieden.")])
# ['Unsere', 'Kund', ':', 'innen', 'und', 'Mitarbeiter', ':', 'innen', 'sind', 'zufrieden', '.']
print([t.text for t in nlp("Wir suchen eine:n Ärzt:in für unser Team.")])
# ['Wir', 'suchen', 'eine', ':', 'n', 'Ärzt', ':', 'in', 'für', 'unser', 'Team', '.']

Expected: Kund:innen, Mitarbeiter:innen, eine:n, Ärzt:in as single
tokens. For comparison, the star and interpunct variants already stay
whole (Kolleg*innen, Expert·innen), only the colon splits, via this
infix in spacy/lang/de/punctuation.py:

r"(?<=[{a}])[:<>=](?=[{a}])".format(a=ALPHA)

The gender colon is the most common separator in German inclusive writing
(the INCLUSIFY benchmark, arXiv:2212.02564, documents frequency across sources). Every downstream component degrades on the split: the noun is tagged in pieces, NER never sees the name-like whole, and lemmas are
computed for the fragments. We created an inclusive language text checker and patch this infix; we'd like to upstream the behavior.

Two possible fixes, happy to submit a PR for either:

  1. Conservative: exempt the noun gender endings from the colon infix -
    r"(?<=[{a}]):(?!in(nen)?\b)(?=[{a}])" - which keeps Kund:in and
    Kund:innen whole. Article forms (eine:n, jede:r) would still
    split; covering them too means exempting short lowercase tails, e.g.
    r"(?<=[{a}]):(?![a-zäöüß]{1,5}\b)(?=[{a}])", at the cost of no longer
    splitting all-lowercase typos like wort:wort.
  2. A tokenizer config option, if changing the default is unwanted.

Your Environment

  • Operating System: macOS-26.6.1-arm64-arm-64bit
  • Python Version Used: 3.12.9
  • spaCy Version Used: 3.8.14
  • Environment Information: de_core_news_lg 3.8.0 (behavior identical on
    3.7.x; long-standing default, not a regression)

Notes

  • The original solution was developed without AI assistance; however, this ticket was written with the help of Claude. I was unable to find an AI policy for this project.
  • Our production patch keeps <>= splitting and drops only the colon.
  • I have another ticket in the pipeline related to ADJA mistagging of colon forms, which is only observable once they survive tokenization.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions