Repository navigation
feat(elastic): make the fulltext analyzer configurable via ELASTIC_ANALYZER - #11147
Open
MichaelUray wants to merge 1 commit into
Open
MichaelUray wants to merge 1 commit into
MichaelUray wants to merge 1 commit into
Conversation
…ALYZER The fulltextSummary field, which holds the extracted text of every indexed document, was always mapped with the custom English analyzer rebuilt_english. Non-English deployments therefore got no stemming, so inflected forms of a word were indexed as unrelated tokens and searches silently missed results. The analyzer is now read from the ELASTIC_ANALYZER environment variable (default: rebuilt_english, so existing deployments are unchanged). Any analyzer known to the cluster can be used, e.g. the built-in russian or german ones. Since Elasticsearch cannot change the analyzer of an existing field, initMapping now also compares the analyzer and recreates the index when it differs instead of failing putMapping in a retry loop; a full reindex is needed afterwards. The variable is documented in pods/fulltext/README.md. Fixes hcengineering#4545 Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4545
Problem and change
The fulltextSummary field, which holds the extracted text of every indexed
document, was always mapped with the custom English analyzer rebuilt_english.
Non-English deployments therefore got no stemming, so inflected forms of a
word were indexed as unrelated tokens and searches silently missed results.
The analyzer is now read from the ELASTIC_ANALYZER environment variable
(default: rebuilt_english, so existing deployments are unchanged). Any analyzer
known to the cluster can be used, e.g. the built-in russian or german ones.
Since Elasticsearch cannot change the analyzer of an existing field,
initMapping now also compares the analyzer and recreates the index when it
differs instead of failing putMapping in a retry loop; a full reindex is
needed afterwards. The variable is documented in pods/fulltext/README.md.
Tests
New unit tests
foundations/server/packages/elastic/src/__tests__/analyzer.test.ts. The default (rebuilt_english) is unchanged. A full check needs a fulltext service with Elasticsearch/OpenSearch: setELASTIC_ANALYZER=russian, restart (the index is recreated), run a reindex, and search with an inflected word.