Skip to content

fix(dedup): normalize SimHash similarity by fingerprint size - #565

Open
BSAIPF wants to merge 1 commit into
OpenDCAI:mainfrom
BSAIPF:fix/simhash-fingerprint-width
Open

BSAIPF wants to merge 1 commit into
OpenDCAI:mainfrom
BSAIPF:fix/simhash-fingerprint-width

Conversation

@BSAIPF

@BSAIPF BSAIPF commented Oct 20, 2026

Copy link
Copy Markdown

SimHashDeduplicateFilter normalizes Hamming distance by len(bin(value)), which includes the 0b prefix and omits leading zero bits. Use the configured Simhash.f width so the same bit distance always produces the same similarity.

With 8-bit fingerprints, world and quality differ by one bit: their similarity is 0.875, but the old calculation returns 0.9 and removes one at bound=0.1. Conversely, hello and seven are incorrectly retained together at bound=0.125 because their leading zeros reduce the denominator.

Add CPU regressions for 8/16/64/128-bit fingerprints, symmetry, identical/opposite values, and both text pairs through JSONL-backed storage using input_key and single-element input_keys.

Validation (Python 3.12):

  • All 8 new cases fail before the fix and pass afterward.
  • pytest -m "cpu or api" -q: 198 passed, 48 deselected.
  • New test file passes Ruff lint and formatting checks; the modified function passes lint, with existing findings elsewhere in its module unchanged.
  • Source distribution and wheel build; twine check --strict passes for both.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant