Meilisearch: v1.4.0 🦓

Meilisearch v1.4 improves the search result relevancy by introducing three new settings: `nonSeparatorTokens`, `separatorTokens` and `dictionary`. 🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment happens between 4 to 48 hours after a new version becomes available. Some SDKs might not include all new features—consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️). # New features and improvements 🔥 ## Customize text separators Meilisearch word segmentation uses a list of characters to separate one word from another. The `nonSeparatorTokens` and `separatorTokens` index settings allow you to configure this to better fit your dataset. Add a character to the `separator-tokens` list to use it as a word separator: ```bash curl \ -X PUT 'http://localhost:7700/indexes/articles/settings/separator-tokens' \ -H 'Content-Type: application/json' \ --data-binary '["&sect;", "&sep"]' ``` Add a character to the `non-separator-tokens` list when you don't want Meilisearch to use it to separate words: ```bash curl \ -X PUT 'http://localhost:7700/indexes/articles/settings/non-separator-tokens' \ -H 'Content-Type: application/json' \ --data-binary '["@", "#", "&"]' ``` Done by @ManyTheFish in #3946 ## Load user-defined dictionaries Meilisearch word segmentation also relies on language-based dictionaries to segment words. Use the `dictionary` index setting to expand the default dictionaries and improve accuracy when working with datasets using domain-specific terms: ```bash curl \ -X PUT 'http://localhost:7700/indexes/articles/settings/dictionary' \ -H 'Content-Type: application/json' \ --data-binary '["J. R. R.", "J.R.R."]' ``` This feature can be used together with the `stopWords` and `synonyms` index settings: ```json { "dictionary": ["J. R. R.", "J.R.R."], "synonyms": { "J.R.R.": ["jrr", "J. R. R."], "J. R. R.": ["jrr", "J.R.R."], "jrr": ["J.R.R.", "J. R. R."], } } ``` Done by @ManyTheFish in #3946 ## Other improvements Meilisearch better hides your data and no longer shows hidden document's fields in error messages. In case you are trying to sort by using a field that is not sortable but there are some of them that are not displayed either, you will see a message like the following: ``` Available sortable attributes are: price, stock, <..hidden-attributes>. ``` Done by @vivek-26 in #3945 # Fixes 🐞 * Fix synonyms display (#3946) @ManyTheFish * Fix thai synonyms (#4033) @Kerollmops * Fix synonyms with separators (#3994) @ManyTheFish * Fix the swap index tasks (#4041) @irevoire * Fix highlighting bug when searching for a phrase with cropping (#4028) @vivek-26 * Empty arrays/objects now return empty instead of null (#3997) @dogukanakkaya * Fix the stats of the documents deletion by filter (#4053) @irevoire * `\` can now be used at the end of the `filter` search parameter values (#4038 and #4043) @Kerollmops ## 🔴 Usage breaking change following `filter` bug fix ⚠️ This section only concerns users using the `filter` search parameter with `\` characters in the filter expression. In this version, we fixed a bug regarding `\` in the `filter` search parameter expression. **Before v1.4.0, the users were not able to express all the filter expression they wanted.** For example, if you add the following documents: ```json [ { "id": 1, "path": "my\\test\\path" }, { "id": 2, "path": "my\\test\\path\\" } ] ``` (`\\` escaping here is about JSON escaping) Before v1.4.0, if you wanted to filter on document 2, the filters `path = "my\\test\\path\\"` and `path = "my\\test\\path\\\"` led to an error. Now, in v1.4.0, you can express all the possible filter expressions, no matter if the filter contains `\` or not. But to reach this, **you have to escape all the `\` characters in your filter**. Following our example, to finally succeed in filtering on document 2, the filter should be: `path = "my\\\\test\\\\path\\\\"`. 👉 If you come from v1.3.X or before, and you used `\` in `filter`, here is the change to apply: following our example, to filter on document 1, before v1.4.0, you could apply the filter `path = "my\\test\\path"`. Now, from v1.4.0, you have to escape the `\` and apply the following filter: `path = "my\\\\test\\\\path"`. Two escapings are applied: the JSON escaping and the Meilisearch filter escaping. The JSON unescaping transforms the `\\\\` into `\\` and the Meilisearch filter unescaping ends up with a single `\`. If you need any help with this change, please feel free to reach us on our [Discord](https://discord.meilisearch.com/). # Misc * Upgrade Meilisearch dependencies (#3987) @ManyTheFish * Upgrade CI dependencies (#3871, #3969 and #3968) * Fix CI benchmarks (#3963) @irevoire * Removed `borrow` call causing failed nightly tests (#3990) @JannisK89 * Expose Puffin server to profile the indexing process (#3913) @Kerollmops * Update and fix the CI Test Suite (#3918) @Kerollmops * Improve CI test suite for manual trigger events (#3989) @curquiza * Update README.md (#3895 and #3932) @ferdi05 & @Strift * Fix Homebrew CI (#4016) @Kerollmops ❤️ Thanks again to our external contributors: - [Meilisearch](https://github.com/meilisearch/meilisearch): @dogukanakkaya, @JannisK89, and @vivek-26.