Meilisearch: v1.12.0 🦗

Meilisearch v1.12 introduces significant indexing speed improvements, almost halving the time required to index large datasets. This release also introduces new settings to customize and potentially further increase indexing speed. 🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment happens between 4 to 48 hours after a new version becomes available. Some SDKs might not include all new features. Consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️). # New features and updates 🔥 ## Improve indexing speed Indexing time is improved across the board! - Performance is maintained or better on smaller machines - On bigger machines with multiple cores and good IO, Meilisearch v1.12 is much faster than Meilisearch v1.11 - More than twice as fast for raw document insertion tasks. - More than x4 as fast for incrementally updating documents in a large database. - Embeddings generation was also improved up to x1.5 for some workloads. The new indexer also makes task cancellation faster. Done by @dureuill, @ManyTheFish, and @Kerollmops in #4900. ## New index settings: use `facetSearch` and `prefixSearch` to improve indexing speed v1.12 introduces two new index settings: `facetSearch` and `prefixSearch`. Both settings allow you to skip parts of the indexing process. This leads to significant improvements to indexing speed, but may negatively impact search experience in some use cases. Done by @ManyTheFish in #5091 ### `facetSearch` Use this setting to toggle [facet search](https://www.meilisearch.com/docs/learn/filtering_and_sorting/search_with_facet_filters#searching-facet-values): ```bash curl \ -X PUT 'http://localhost:7700/indexes/books/settings/facet-search' \ -H 'Content-Type: application/json' \ --data-binary 'true' ``` The default value for `facetSearch` is `true`. When set to `false`, this setting disables facet search for all filterable attributes in an index. ### `prefixSearch` Use this setting to configure the ability to [search a word by prefix](https://www.meilisearch.com/docs/learn/engine/prefix) on an index: ```bash curl \ -X PUT 'http://localhost:7700/indexes/books/settings/prefix-search' \ -H 'Content-Type: application/json' \ --data-binary 'disabled' ``` `prefixSearch` accepts one of the following values: - `"indexingTime"`: enables prefix processing during indexing. This is the default Meilisearch behavior - `"disabled"`: deactivates prefix search completely Disabling prefix search means the query `he` will no longer match the word `hello`. This may significantly impact search result relevancy, but speeds up the indexing process. ## New API route: `/batches` The new `/batches` endpoint allow you to query information about task batches. `GET` `/batches` returns a list of batch objects: ```sh curl -X GET 'http://localhost:7700/batches' ``` This endpoint accepts the same parameters as `GET` `/tasks` route, allowing you to narrow down which batches you want to see. Parameters used with `GET` `/batches` apply to the tasks, not the batches themselves. For example, `GET /batches?uid=0` returns batches containing tasks with a `taskUid` of `0` , not batches with a `batchUid` of `0`. You may also query `GET` `/batches/:uid` to retrieve information about a single batch object: ```sh curl -X GET 'http://localhost:7700/batches/BATCH_UID' ``` `/batches/:uid` does not accept any parameters. Batch objects contain the following fields: ```json5 { "uid": 160, "progress": { "steps": [ { "currentStep": "processing tasks", "finished": 0, "total": 2 }, { "currentStep": "indexing", "finished": 2, "total": 3 }, { "currentStep": "extracting words", "finished": 3, "total": 13 }, { "currentStep": "document", "finished": 12300, "total": 19546 } ], "percentage": 37.986263 }, "details": { "receivedDocuments": 19547, "indexedDocuments": null }, "stats": { "totalNbTasks": 1, "status": { "processing": 1 }, "types": { "documentAdditionOrUpdate": 1 }, "indexUids": { "mieli": 1 } }, "duration": null, "startedAt": "2024-12-12T09:44:34.124726733Z", "finishedAt": null } ``` Additionally, task objects now include a new field, `batchUid`. Use this field together with `/batches/:uid` to retrieve data on a specific batch. ```json5 { "uid": 154, "batchUid": 142, "indexUid": "movies_test2", "status": "succeeded", "type": "documentAdditionOrUpdate", "canceledBy": null, "details": { "receivedDocuments": 1, "indexedDocuments": 1 }, "error": null, "duration": "PT0.027766819S", "enqueuedAt": "2024-12-02T14:07:34.974430765Z", "startedAt": "2024-12-02T14:07:34.99021667Z", "finishedAt": "2024-12-02T14:07:35.017983489Z" } ``` Done by @irevoire in #5060, #5070, #5080 ## Other improvements * New query parameter for `GET` `/tasks`: `reverse`. If `reverse` is set to `true`, tasks will be returned in reversed order, from oldest to newest tasks. Done by @irevoire in #5048 * Phrase searches with`showMatchesPosition` set to `true` give a single location for the whole phrase @flevi29 in #4928 * New Prometheus metrics by @PedroTurik in #5044 * When a query finds matching terms in document fields with array values, Meilisearch now includes an `indices` field to `_matchesPosition` specifying which array elements contain the matches by @LukasKalbertodt in #5005 * ⚠️ Breaking `vectorStore` change: field distribution no longer contains `_vectors`. Its value used to be incorrect, and there is no current use case for the fixed, most likely empty, value. Done as part of #4900 * Improve error message by adding index name in #5056 by @airycanon # Fixes 🐞 * Return appropriate error when primary key is greater than 512 bytes, by @flevi29 in #4930 * Fix issue where numbers were segmented in different ways depending on tokenizer, by @dqkqd in https://github.com/meilisearch/charabia/pull/311 * Fix pagination when embedding fails by @dureuill in https://github.com/meilisearch/meilisearch/pull/5063 * Fix issue causing Meilisearch to ignore stop words in some cases by @ManyTheFish in #5062 * Fix phrase search with `attributesToSearchOn` in #5062 by @ManyTheFish # Misc * Dependencies updates * Update benchmarks to match the new crates subfolder by @Kerollmops in #5021 * Fix the benchmarks by @irevoire in #5037 * Bump Swatinem/rust-cache from 2.7.1 to 2.7.5 in #5030 * Update charabia v0.9.2 by @ManyTheFish in #5098 * Update mini-dashboard to v0.2.16 version by @curquiza in #5102 * CIs and tests * Improve performance of `delete_index.rs` by @DerTimonius in #4963 * Improve performance of `create_index.rs` by @DerTimonius in #4962 * Improve performance of `get_documents.rs` by @PedroTurik in #5025 * Improve performance of `formatted.rs` by @PedroTurik in #5043 * Fix the path used in the flaky tests CI by @Kerollmops in #5049 * Misc * Rollback the Meilisearch Kawaii logo by @Kerollmops in #5017 * Add image source label to Dockerfile by @wuast94 in #4990 * Hide code complexity into a subfolder by @Kerollmops in #5016 * Internal tool: implement offline upgrade from v1.10 to v1.11 by @irevoire in #5034 * Internal tool: implement offline upgrade from v1.11 to v1.12 by @ManyTheFish in #5146 * Meilisearch is now able to retrieve Katakana words from a Hiragana query by @tats-u in https://github.com/meilisearch/charabia/pull/312 * Improve error handling when writing into LMDB by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/5089 ❤️ Thanks again to our external contributors: - [Meilisearch](https://github.com/meilisearch/meilisearch): @airycanon, @DerTimonius, @flevi29, @LukasKalbertodt, @PedroTurik, @wuast94 - [Charabia](https://github.com/meilisearch/charabia): @dqkqd @tats-u