Meilisearch: v1.40.0 🦈

This release introduced support for the `distinct` attribute in federated search, enabling cross-index distinct attributes with facet distribution support. Additionally, significant performance improvements were delivered, including faster federated search (approximately 100ms faster), optimized JSON document generation for better handling of large documents and a much better memory usage for large workloads. ### ✨ Enhancement - **Support `distinct` in federated search** by @dureuill in https://github.com/meilisearch/meilisearch/pull/6214 The `distinct` attribute can now be passed to the `federation` object in federated search to apply a global, cross-index and cross-remote distinct computation to the results. <details> <summary>Example of a federated search request with distinct</summary> ```json { "federation": { "distinct": "genres", // ✨ NEW "facetsByIndex": { // recovering facet distribution is also supported with distinct "comics": [ "genres" ], "movies": [ "genres" ] }, "mergeFacets": {} // merging facet distributions is also supported with distinct }, "queries": [ { "indexUid": "comics", "q": "batman", "attributesToRetrieve": ["title", "genres"], "useNetwork": true // distinct is also supported with network queries }, { "indexUid": "movies", "q": "superman", "attributesToRetrieve": ["title", "genres"], "useNetwork": true } ] } ``` </details> <details> <summary>Sample response to a federated search request with distinct</summary> ```json { "hits": [ { "title": "Batman", "genres": [ "Family", "Adventure", "Comedy", "Science Fiction", "Crime" ], "_federation": { "indexUid": "comics", "queriesPosition": 0, "weightedRankingScore": 1.0, "remote": "ms2" } }, { "title": "Batman", "genres": [ "Fantasy", "Action" ], "_federation": { "indexUid": "comics", "queriesPosition": 0, "weightedRankingScore": 1.0, "remote": "ms1" } }, { "title": "Batman & Bill", "genres": [ "Documentary" ], "_federation": { "indexUid": "comics", "queriesPosition": 0, "weightedRankingScore": 0.9848484848484848, "remote": "ms1" } }, { "title": "Superman: Red Son", "genres": [], "_federation": { "indexUid": "movies", "queriesPosition": 1, "weightedRankingScore": 0.9848484848484849, "remote": "ms0" } }, { "title": "Superman, Spider-Man or Batman", "genres": [ "Drama" ], "_federation": { "indexUid": "movies", "queriesPosition": 1, "weightedRankingScore": 0.9848484848484849, "remote": "ms0" } } ], "processingTimeMs": 15, "limit": 5, "offset": 0, "estimatedTotalHits": 11, "facetDistribution": { "genres": { "Action": 1, "Adventure": 1, "Comedy": 3, "Crime": 2, "Documentary": 1, "Drama": 1, "Family": 1, "Fantasy": 1, "Horror": 2, "Romance": 1, "Science Fiction": 1, "Thriller": 1, "Western": 1 } }, "facetStats": {}, "requestUid": "019d05c7-ea65-77a1-8274-22a8ba9e26db", "remoteErrors": {} } ``` </details> Note the following to apply the distinct attribute at the federation level: 1. Applying `distinct` at the query level at the same time as the federation level is disallowed and will return a HTTP 400 error. 2. The chosen distinct field will apply to all remotes and indexes, so it must be a filterable attribute for all participating remotes and indexes. 3. While Meilisearch attempts to compute the most accurate facet distribution, in distributed contexts this cannot be guaranteed as the distinct algorithm is not applied on all of the remote documents. - **Improve performance of federated search** by @dureuill in https://github.com/meilisearch/meilisearch/pull/6229 Improves performance of federated search: about 100ms faster for all requests. Improves reliability of the HTTP server: the server will no longer be blocked when too many federated search requests are being processed. - **Optimize the generation of JSON documents** by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6257 Addresses performance issues that users might encounter when requesting large documents. Additionally, performance is enhanced when users request only a small subset of fields from large documents. - **Use the latest version of mimalloc to improve memory usage** by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6201 Updates mimalloc from v2 to v3, improving memory sharing between threads and significantly reducing memory usage on large workloads. It also overrides the allocator to use mimalloc at linking time, allowing LMDB, Meilisearch, and other C libraries to share their allocations for better overall memory efficiency. @Kerollmops wrote [a blog post about the story behind this improvement](https://blog.kerollmops.com/the-good-the-bad-and-the-leaky-jemalloc-bumpalo-and-mimalloc-in-meilisearch). - **Add `POST /tasks/compact` for task queue compaction** by @YoEight in https://github.com/meilisearch/meilisearch/pull/6193 Compacts the task queue database and reclaim space so new tasks can keep being enqueued, without deleting existing tasks. This feature is behind the `taskQueueCompactionRoute` experimental feature flag. > [!WARNING] > Once task queue compaction completes, all write operations are blocked until the server is restarted. ### 🔐 Security - Bump rustls-webpki from 0.103.8 to 0.103.10 in https://github.com/meilisearch/meilisearch/pull/6273 - Bump tar from 0.4.44 to 0.4.45 in https://github.com/meilisearch/meilisearch/pull/6271 - Bump actix-web-lab from 0.24.3 to 0.26.0 in https://github.com/meilisearch/meilisearch/pull/6218 - Bump quinn-proto from 0.11.13 to 0.11.14 in https://github.com/meilisearch/meilisearch/pull/6217 ### 🪲 Bug fixes - Check char boundaries when hiding secrets by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6262 - Network fixes by @dureuill in https://github.com/meilisearch/meilisearch/pull/6255 - Override "default" source base url with the config one to support AWS bedrock OpenAI API by @dureuill in https://github.com/meilisearch/meilisearch/pull/5905 - Fix panic on virtiofs filesystem by @itsamine27 in https://github.com/meilisearch/meilisearch/pull/6261 ### 🔩 Miscellaneous - Add `X-Accel-Buffering: no` for `/logs/stream` route by @YoEight in https://github.com/meilisearch/meilisearch/pull/6268 - Add link to documentation on `--experimental-allowed-ip-networks` parameter by @paulden in https://github.com/meilisearch/meilisearch/pull/6263 - Update sysinfo version to `0.38.4` by @GuillaumeGomez in https://github.com/meilisearch/meilisearch/pull/6211 - Add a no-agent [AGENTS.md](http://agents.md/) file to prevent excessive AI-generated PRs by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6256 ## New Contributors - @GuillaumeGomez made their first contribution in https://github.com/meilisearch/meilisearch/pull/6211 - @itsamine27 made their first contribution in https://github.com/meilisearch/meilisearch/pull/6261