Meilisearch: v1.30.0 🐸

# 🌈 Improvements ## Up and down network scaling Since [v1.19.0](https://github.com/meilisearch/meilisearch/releases/tag/v1.19.0), Meilisearch [Enterprise Edition](https://github.com/meilisearch/meilisearch/blob/main/README.md#-enterprise-edition-ee) allows the automatic sharding of documents over multiple Meilisearch instances, enabling scaling to more documents than a single instance would accommodate. Meilisearch v1.30.0 adds the ability to modify the number of participants in sharding, without having to start over sending documents to a new cluster containing the number of desired machines. To make this possible, Meilisearch v1.30.0 introduces _breaking changes_. These breaking changes only affect the users of the experimental `network` feature who enabled automatic sharding (`network.sharding = true`). Users of the stable features are not affected. ### Usage <details> <summary>Setting up the initial network</summary> 1. Pick a leader machine that will receive all tasks. Calling a route that creates a dcument, settings or network-related task on a machine that is not the leader will now return an error. 2. Send your network topology to the leader by calling `PATCH /network`: ```jsonc // PATCH /network { "self": "ms0", "leader": "ms0", // must be equal to `self` "remotes": { "ms0": { "url": "URL_OF_MS0", "searchApiKey": "SEARCH_API_KEY_OF_MS0", "writeApiKey": "WRITE_API_KEY_OF_MS0", }, "ms1": { "url": "URL_OF_MS1", "searchApiKey": "SEARCH_API_KEY_OF_MS1", "writeApiKey": "WRITE_API_KEY_OF_MS1", } } } ``` 3. The network is automatically propagated to other members of the network. 4. Send settings and documents as usual, but exclusively to the leader. They will be propagated to all participants in the network, and each participant will process a piece of the documents. </details> <details> <details> <summary>Upgrading from v1.29 or earlier</summary> 1. We recommend using the experimental dumpless upgrade feature to go from Meilisearch v1.13+ to Meilisearch v1.29. 2. When using the experimental dumpless upgrade and the Meilisearch instance already has a `network` instance with `sharding` set to `true`, then the `network` object will be modified so that the leader is the first remote in alphabetic order. For instance, if you network contains remotes: `A`, `B`, `C`, the leader will be set to `A`. </details> <summary>Adding a new remote</summary> 1. Call `PATCH /network` on the leader with the information about the new remote: ```jsonc { "remotes": { // add information about the new remote "ms2": { "url": "URL_OF_MS2", "searchApiKey": "SEARCH_API_KEY_OF_MS2", "writeApiKey": "WRITE_API_KEY_OF_MS2", } // information about existing remotes does not need to be repeated } ``` 4. The new network will be propagated from the leader to all remotes (including the new remote `ms2`). 5. All remotes will register a new `networkTopologyChange` task that will "rebalance" the documents between the existing remotes and the new remote, that is, `ms0` and `ms1` will send parts of their documents to `ms2` </details> <details> <summary>Removing a remote</summary> 1. Call `PATCH /network` on the leader by setting any removed remote to `null`: ```jsonc { "remotes": { // set removed remote to null "ms2": null // information about existing remotes does not need to be repeated } ``` 2. The new network will be propagated from the leader to all remotes (including to the old remote `ms2` that will then no longer participate in the network). 3. All remotes will register a new `networkTopologyChange` task that will "rebalance" the documents between the remaining remotes, that is, `ms2` will send its documents between `ms0` and `ms1` </details> ### List of changes <details> <summary>List of changes, some of which are breaking</summary> 1. Breaking change: The `Network` object returned or edited by the `/network` route is modified in the following way: - the `sharding` boolean is removed - a `leader` field is added as an optional string. When it is not `null`, it has the same effect (and more) than having `sharding: true` in the previous iteration of the `Network` object. The leader is used as a check when receiving task creation requests. - a `version` field is added as a UUID, defaulting to the null UUID. The version is used when processing tasks. 6. Breaking change: When a `network.leader` is set, calling one of the following routes will fail with `not_a_leader` error if the target's `network.self` is not the same as its `network.leader`: - `POST /indexes` - `PATCH/DELETE /indexes/{:indexUid}` - `POST/PUT/DELETE /indexes/{:indexUid}/documents` - `POST /indexes/{:indexUid}/documents/delete` - `POST /indexes/{:indexUid}/documents/delete-batch` - `POST /indexes/{:indexUid}/documents/edit` - `PATCH/DELETE /indexes/{:indexUid}/settings` and settings sub-routes - `PATCH /network` if the target is the **new** leader - `POST /swap-indexes` 7. Breaking change: when a `leader` is set, `PATCH /network` no longer returns a `Network` object. Rather, it spawns a new `NetworkTopologyChange` task, and returns the summarized task view. 8. Breaking change: Tasks are duplicated by the leader to the entire network when calling the following routes: - `POST /indexes` (new to this PR) - `PATCH/DELETE /indexes/{:indexUid}` (new to this PR) - `POST/PUT/DELETE /indexes/{:indexUid}/documents` (was already the case before this PR) - `POST /indexes/{:indexUid}/documents/delete` (was already the case before this PR) - `POST /indexes/{:indexUid}/documents/delete-batch` (was already the case before this PR) - `POST /indexes/{:indexUid}/documents/edit` (was already the case before this PR) - `PATCH/DELETE /indexes/{:indexUid}/settings` and settings sub-routes (new to this PR) - `PATCH /network` if the target is the **new** leader (new to this PR) - `POST /swap-indexes` (new to this PR) 9. New `NetworkTopologyChange` tasks that perform the following: 1. Execute any remaining task to process with a `network.version` lower than the network task's version 6. Iterate over all documents in all indexes, determine their new shard, and send the document to the remote that must now have it in the new version , deleting it from the local DB - The export route code has been factored and specialized to allow this - Should the export to a remote fail, the corresponding documents are kept locally - If there are no documents to send for an index, still call the documents addition with an empty payload and appropriate headers containing the expected metadata - If there are no documents to send for an entire remote, call the `network` route with specific headers containing the expected metadata 7. Wait for and process tasks from the remotes of the previous version of the network. 10. Breaking change: When importing dumps, we drop the `self` and `leader` from the network 11. Network topology change tasks can be cancelled. In this case the state will be the current one (any moved documents will stay that way). Cancellation needs to happen on all machines. </details> by @dureuill in https://github.com/meilisearch/meilisearch/pull/6000 ## 🛠️ Maintenance - Fix macos-amd64 compilation by @dureuill in https://github.com/meilisearch/meilisearch/pull/6037 - Move to Rust v1.91.1 by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6039 - Remove risk of command injection in CI by @curquiza in https://github.com/meilisearch/meilisearch/pull/6041 ## Asset availability note 🌈 The Meilisearch binary is available again for `meilisearch-enterprise-macos-amd64` and `meilisearch-macos-amd64`. It was not available for Meilisearch v1.29. **Full Changelog**: https://github.com/meilisearch/meilisearch/compare/v1.29.0...v1.30.0