Meilisearch: v1.2.0 🐻‍❄️

Meilisearch v1.2 introduces new filter operators `IS NULL` and `IS EMPTY`, a new route to delete documents with filters, and significant relevancy and performance updates. 🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment happens between 4 to 48 hours after a new version becomes available. Some SDKs might not include all new features—consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️). # New features and improvements 🔥 ## Delete documents by filter Meilisearch v1.2 allows you to use filters to delete documents with the new `/documents/delete` route: ```bash curl -X POST http://localhost:7700/indexes/dogs/documents/delete \ -H 'Content-Type: application/json' \ --data-binary '{ "filter": ["doggo = 'bernese mountain'", "face = cute"] }' ``` Fields must be set as [filterable](https://www.meilisearch.com/docs/reference/api/settings#update-filterable-attributes) before you can use them as filters. Meilisearch will return a summarized task object: ```json { "taskUid": 242, "indexUid": "dogs", "status": "enqueued", "type": "documentDeletion", "enqueuedAt": "2023-05-03T11:01:58.721841Z" } ``` Use the returned `taskUid` with the [task route](https://www.meilisearch.com/docs/reference/api/tasks#get-one-task) to check the task status. Done by @dureuill and @irevoire in #3550. ## Get documents by filter You can now use filters in the `GET` endpoint of the `/documents` route: ```bash curl -X GET 'http://localhost:7700/indexes/dogs/documents?limit=1&filter=doggo=bernese' ``` You can also use the new `/documents/fetch` route to handle complex filters: ```bash curl -X POST http://localhost:7700/indexes/dogs/documents/fetch \ -H 'Content-Type: application/json' \ --data-binary '{ "limit": 1, "filter": "doggo = bernese" }' ``` `/documents/fetch` accepts following parameters: `limit`, `offset`, `fields`, and `filter`. Fields must be set as [filterable](https://www.meilisearch.com/docs/reference/api/settings#update-filterable-attributes) before you can use them as filters. Done by @dureuill and @irevoire in #3570. ## New filter operators: `IS EMPTY` and `IS NULL` This release introduces two new filter operators. `IS EMPTY` matches existing fields with a valid, but empty value. `IS NULL` matches existing fields with an explicit `null` value. Given the following documents: ```json [{ "id": 0, "color": [] }, { "id": 1, "color": null }, { "id": 2, }, ] ``` `color IS EMPTY` matches document `0`. `color IS NULL` matches document `1`. Both operators can be used together with the `NOT` operator: `color IS NOT EMPTY` and `NOT color IS EMPTY` match document `1`. `color IS NOT NULL` and `NOT color IS NULL` match document `0`. Neither operator will match documents missing the specified field. Done by @Kerollmops in #3571. ## Improve relevancy and search performance The search engine has seen significant refactoring. This brings performance and relevancy improvements, as well as setting the foundations for new features related to search customization. Stay tuned! If you have any questions or experience unexpected behavior, feel free to [reach out](https://github.com/meilisearch/meilisearch/discussions). Done by @loiclec, @dureuill, @ManyTheFish, @Kerollmops and @irevoire in #3542. ### Performance improvements - The fastest 75 percent of queries now consistently answer below 50ms on our test setup - Complexity of search queries has been limited to limit CPU time and RAM consumption. More specifically: - A single term is now limited to 150 possible typo matches for queries containing 1 typo, and to 50 possibilities for queries containing 2 typos - Both single word or multi word queries consider a maximum of 50 synonyms - The total number of words for all synonyms of a single term cannot exceed 100 - Queries can now contain a maximum of 1000 words. This change only impacts queries containing very long phrases. Search terms (words not inside a phrase, and individual phrases) remain limited to 10 per query - Geo search improvements: - Increased performance when geosorting small sets of documents - Descending sort is now as performant as ascending sort - Address assorted minor performance issues: https://github.com/meilisearch/meilisearch/issues/3124, https://github.com/meilisearch/meilisearch/issues/3378, https://github.com/meilisearch/meilisearch/issues/2202, https://github.com/meilisearch/meilisearch/issues/3123 ### Relevancy improvements - The `exactness` ranking rule no longer treats synonyms as exact matches. This boosts documents containing the query exactly as typed by the user. Solves https://github.com/meilisearch/meilisearch/issues/3410 - Meilisearch will always sort results as if the `words` ranking rule were present with a higher priority than the `attributes`, `exactness`, `typo` and `proximity` ranking rules. This happens even if the `words` ranking rule has been removed or set with a lower priority. Search behavior when using the specified ranking rules before or without `words` was confusing and hurt relevancy - Split words are now also treated as possible digrams. This means the query `whit ehorse` may match documents containing `white horse`, which improves the relevancy of typo tolerance - N-grams and split words are now ranked lower in comparison to the exact word when processing the `typo` ranking rule. For example, a query for `sun flower` will now return a document titled `Sun Flower` before one titled `Sunflower` - Ranking rule behavior no longer depends on the number of ranked documents, improving consistency in terms of both relevancy and latency. Solves https://github.com/meilisearch/meilisearch/issues/3356 ## Automated task deletion Maximum number of tasks the task queue can hold now limited to 1M. Fixes issues with enqueuing new tasks when the database is full. Once the task queue reaches this limit, Meilisearch will try to delete the oldest 100k tasks. If not all 100k tasks can be deleted due to some of them being unfinished, Meilisearch will delete as many tasks as possible. If there are no unfinished tasks, Meilisearch will not delete anything but still enqueue new tasks as usual. The engine will stop you from adding new tasks once the hard limit of 10GiB of tasks is reached (that’s between 5M and 15M of tasks depending on your workflow). In this case, Meilisearch will try to automatically delete unfinished tasks. If this is unsuccessful, it will not enqueue new tasks, and log a warning notifying the user the engine is not working properly. Done by @irevoire in #3693. ## Language support improvements * Split camelCase in Latin segmenter * Improve Arabic: normalizer and segmentation Done by @ManyTheFish, @akeamc, @DrAliRagab, @goodhoko, and @mosuka in #3702 and in [Charabia v0.7.2](https://github.com/meilisearch/charabia/releases/tag/v0.7.2) ## Other improvements * Introduce experimental feature to reduce the RAM usage. Can be useful if Meilisearch is sometimes killed by your OS. [Check out this discussion](https://github.com/meilisearch/product/discussions/652) to get more information and how to use it (#3651) @Kerollmops * Add boolean support for CSV documents (#3576) @irevoire * Improve geosort errors during sorting and filtering (#3638 and #3631) @cymruu * Improve error message when payload is too large (#3739) @cymruu # Fixes 🐞 * Improve the `GET /health` route by ensuring the internal DB is not down (#3647) @irevoire * Handle single quotation mark (`’`) and apostrophe (`'`) the same way (#3702) @ManyTheFish * Fix the sort error message: regression seen in v1.1 (#3749 and #3755) @ManyTheFish # Misc * Dependencies upgrade * Bump the Meilisearch dependencies (#3661) @Kerollmops * Security: bump h2 from 0.3.15 to 0.3.17 (#3674) * Update clippy toolchain from v1.67 to v1.69 (#3695) @curquiza * Bump GitHub actions dependencies: #3554, #3551, #3552, #3553, #3633 * CIs and tests * Add scheduled tests for all features to Actions (#3510) @jlucktay * Clean CI file names (#3567) @curquiza * Enable cache again in test suite CI (#3587) @curquiza * Remove CLI changes for clippy (#3464) @dureuill * Improve docker cache in CIs (#3566) @inductor * Remove Uffizzi because the team no longer uses it (#3694) @curquiza * Add SDK tests to CI (#3709) @curquiza * Use URL of our self-hosted bors instance (#3721) @curquiza * Improve index-scheduler tests (#3639) @Sufflope * Documentation * Add newline after the meilisearch version in the issue template (#3636) @bidoubiwa * Add sprint issue to template issues (#3643) @curquiza * Update the prototype section in CONTRIBUTING.md (#3649) @curquiza * Update README to reference new docs website (#3666, #3710, #3718 and #3720) @guimachiavelli @roy9495 @Kerollmops @curquiza * Move comments above keys in `config.toml` (#3731) @jirutka * Misc * Milli scope: when updating settings, check if primary key actually changes before throwing an error (#3608) @GregoryConrad * Remove unused snapshot files (#3696) @irevoire * Add missing analytics (#3738) @irevoire ❤️ Thanks again to our external contributors: - [Meilisearch](https://github.com/meilisearch/meilisearch): @cymruu, @GregoryConrad, @inductor, @jirutka, @jlucktay, @roy9495, and @Sufflope. - [Charabia](https://github.com/meilisearch/charabia): @akeamc, @DrAliRagab, @goodhoko, and @mosuka.