Meilisearch: v1.13.0 🕊️

Meilisearch v1.13 stabilizes AI-powered search integration! 🎉 This release also introduces two new experimental features: upgrading to new Meilisearch releases without generating a dump and making federated requests across multiple Meilisearch instances. <!-- The following lines should NOT be put in the PRE-release changelogs --> 🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment takes 4 to 48 hours after a new version becomes available. <!-- The following lines should NOT be put in the PRE-release changelogs --> Some SDKs might not include all new features. Consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️). # New features and updates 🔥 ## AI-powered search and vector store stabilization With v1.13, AI-powered search is available to all users by default and no longer requires manual activation. ### Breaking Changes * `vectorStore` is no longer an accepted value for the `/experimental-features` route * Ollama URLs must end with either `/api/embed` or `/api/embeddings` * Modified error codes: - `invalid_embedder` has been split into `invalid_search_embedder` and `invalid_similar_embedder`. These codes are returned when the `embedder` parameter of a request to `/search`, `/multi-search` or `/similar` specifies a non-existing embedder or is not a string - `invalid_hybrid_query` has been renamed to `invalid_search_hybrid_query`. It is returned when the `hybrid` parameter contains unknown keys or is not either `null` or an object Done by @dureuill in https://github.com/meilisearch/meilisearch/pull/5232 & https://github.com/meilisearch/meilisearch/pull/5234 ## Experimental feature: Dumpless upgrades Use `--experimental-dumpless-upgrade` after updating the Meilisearch binary to migrate to a new release: ```sh ./meilisearch --experimental-dumpless-upgrade ``` This faster and more efficient process does not require you to generate a dump and pass it to Meilisearch when upgrading to a new release. > [!WARNING] > Meilisearch strongly recommends you generate a backup snapshot before migrating. This is an experimental feature, and failed upgrades may lead to database corruption. Read more about it [on the dedicated product discussion](https://github.com/orgs/meilisearch/discussions/804). Done by @irevoire and @dureuill in https://github.com/meilisearch/meilisearch/pull/5264 ## Experimental feature: Remote federated search requests Use remote federated search requests together with the `/multi-search` route to query multiple Meilisearch instances simultaneously. This is particularly useful when handling very large databases. First, use the `/experimental-features` route to enable `network`: ```sh curl \ -X PATCH 'MEILISEARCH_URL/experimental-features/' \ -H 'Content-Type: application/json' \ --data-binary '{ "network": true }' ``` Next, set up your network of Meilisearch instances with a call to `PATCH /network` configuring one `self` and multiple `remotes`: ```sh curl \ -X PATCH 'MEILISEARCH_URL/network/' \ -H 'Content-Type: application/json' \ --data-binary '{ "remotes": { "ms-0": { "url": "http://ms-1235.example.meilisearch.io", "searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas" }, "ms-1": { "url": "http://ms-4242.example.meilisearch.io", "searchApiKey": "hrVu-OMcjPGElK7692K7bwriBoGyHXTMvB5NmZkMKqQ" } }, "self": "ms-0" }' ``` Repeat this process with every instance in your network, then add documents. Do not send the same documents to different instances. Finally, make a `/multi-search` query with the new `federationOptions.remote`: ```sh curl \ -X PATCH 'MEILISEARCH_URL/multi-search/' \ -H 'Content-Type: application/json' \ --data-binary '{ "federation": {}, "queries": [ { "q": "Batman returns dark", "indexUid": "movies", "federationOptions": { "remote": "ms-0" } }, { "q": "Batman returns dark", "indexUid": "movies", "federationOptions": { "remote": "ms-1" } } ] }' ``` Find more information about this feature on the [public usage page](https://meilisearch.notion.site/API-usage-Remote-search-request-f64fae093abf409e9434c9b9c8fab6f3?pvs=74) ## Other improvements * Improve task auto-batching by exposing a way to limit the total size of a batch by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/5223 * Add Prometheus metrics to measure task queue latency by @takaebato in https://github.com/meilisearch/meilisearch/pull/5178 * Speed up the request to list indexes by @irevoire in https://github.com/meilisearch/meilisearch/pull/5166 * Add support for `GITHUB_TOKEN` authentication in installation script by @Sherlouk in https://github.com/meilisearch/meilisearch/pull/5216 * Expose a new `usedDatabaseSize` field on the `/stats` route by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/5314 * Expose embeddings info on the `/stats` route by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/5341 # Fixes 🐞 * Improve error message when an attribute is not filterable by @jameshiew in https://github.com/meilisearch/meilisearch/pull/5135 * Make sure indexing embeddings is using the right number of threads by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/5322 * Dumpless upgrade will no longer corrupt the database in case of early stop by @irevoire in https://github.com/meilisearch/meilisearch/pull/5316 * Fix export/import dump that contains batches data by @irevoire in https://github.com/meilisearch/meilisearch/pull/5310 * Make sure indexing geo points works with PUT requests by @dureuill in https://github.com/meilisearch/meilisearch/pull/5332 * Add back semantic search timeout from v1.11.3 by @dureuill in https://github.com/meilisearch/meilisearch/pull/5339 # Misc * Dependency updates * Upgrade dependencies to fix the `idna` severity issue by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/5218 * CIs and tests * Split tests into separate files by @K-Kumar-01 in https://github.com/meilisearch/meilisearch/pull/5134, https://github.com/meilisearch/meilisearch/pull/5171, https://github.com/meilisearch/meilisearch/pull/5174 * Remove obsolete test code by @K-Kumar-01 in https://github.com/meilisearch/meilisearch/pull/5173 * Fix flaky batches test by @irevoire in https://github.com/meilisearch/meilisearch/pull/5175 * Remove hard coded task IDs to prevent flaky tests by @mhmoudr in https://github.com/meilisearch/meilisearch/pull/5182 * Improve test performance of get_index.rs by @DerTimonius in https://github.com/meilisearch/meilisearch/pull/5210 * Other * Instruct users to create custom reports on the benchboard by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/5029 * Fix typo in a comment by @eltociear in https://github.com/meilisearch/meilisearch/pull/5184 * Replace hardcoded string with constants by @Gnosnay in https://github.com/meilisearch/meilisearch/pull/5169 * Refactor index-scheduler by @irevoire in https://github.com/meilisearch/meilisearch/pull/5199 * Refactor indexer by @dureuill in https://github.com/meilisearch/meilisearch/pull/5168 * Auto-generate OpenAPI spec by @irevoire in https://github.com/meilisearch/meilisearch/pull/4867 & https://github.com/meilisearch/meilisearch/pull/5231 * Merge bitmaps by using `Extend::extend` by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/5221 * Fix issue introduced by dumpless upgrade by @manojks1999 in https://github.com/meilisearch/meilisearch/pull/5284 * Send the OSS analytics once per day instead of once per hour by @irevoire in https://github.com/meilisearch/meilisearch/pull/5312 * Log more metrics around HTTP embedder requests and ANNs in arroy by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/5288 * Fix internal error when not correctly using the `documentTemplateMaxBytes` parameter by @dureuill in https://github.com/meilisearch/meilisearch/pull/5306 * When a batch is deleted it no longer keeps some internal data in the DB @Kerollmops and @irevoire in https://github.com/meilisearch/meilisearch/pull/5272 * Fix a rare stack overflow when using remote embedders by @Kerollmops with the help of @dureuill https://github.com/meilisearch/meilisearch/pull/5294 (from v1.12.8) * Fix Dotnet tests in sdks-tests.yml by @curquiza in https://github.com/meilisearch/meilisearch/pull/5291 * Debug log the channel congestion by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/5177 * Improve unexpected panic message by @irevoire in https://github.com/meilisearch/meilisearch/pull/5300 * Introduce a compaction subcommand in meilitool by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/5235 * Expose a route to get the file content associated with a task by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/5326 * Fix workload sha for the benchmark by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/5342 ❤️ Thanks again to our external contributors: - [Meilisearch](https://github.com/meilisearch/meilisearch): @takaebato, @Sherlouk, @jameshiew, @K-Kumar-01, @mhmoudr, @eltociear, @Gnosnay, @DerTimonius, @manojks1999, @ellnix, @Guikingone, @migueltarga