Meilisearch: v1.0.0 🐝

🎉 🎉 We are proud to announce the first major release of Meilisearch! 🎉 🎉 This new release focuses on stabilizing the CLI instead of only introducing new features. We took this opportunity to introduce final breaking changes to the CLI and make the error handler more intuitive. There will be no breaking changes in future versions until the release of v2.0.0. Please refer to our [versioning policy](https://github.com/meilisearch/engine-team/blob/main/resources/versioning-policy.md) for a definition of what we consider to be a breaking change in Meilisearch. Despite our focus on stabilizing Meilisearch, we also took the time to improve support for Chinese and Korean! 🧰 Most integrations (SDKs, clients, libraries) will be compatible with this version within four hours of the release. Sometimes this can take up to 48 hours, depending on the issues we encounter during the release. Here is the exhaustive list of integrations **not immediately compatible** with v1.0.0: - [meilisearch-swift](https://github.com/meilisearch/meilisearch-swift): though you can still use this library with v1.0.0, some new features present in Meilisearch v0.30.0 and later are not yet available. Let us know if you really need it—and contributions are always welcome in the related repository! - [meilisearch-java](https://github.com/meilisearch/meilisearch-java): our team is still working on refactoring the code; this is reaching the end, but the SDK cannot be used with v0.29.0 and later yet. Follow the repository to know when it will be compatible. # New features and improvements 🔥 ## Improve language support * Add support for Korean 🇰🇷 * Improve support for Chinese: * Normalize Chinese characters into Pinyin * Disable the HMM (Hidden Markov Model algorithm) feature in Chinese segmentation * Unify Chinese characters variants (kvariants, zvariants, semantic variants, etc) * Enhance Hebrew, Thai, Arabic, and Latin languages support by improving diacritics and non-spacing marks normalization Done by @ManyTheFish, @dureuill, @choznerol, @crudiedo, @daniel-shuy, @harshalkhachane, @mosuka, @qbx2, @Roms1383, @Sokom141, and @yenwel in #3269 and meilisearch/milli#736 and all the Charabia's PRs listed [here](https://github.com/meilisearch/charabia/releases/tag/v0.7.0). ## Indexing and search speed improvements * Significantly improve the memory usage of search requests containing multiple long words (#3269 and meilisearch/milli#708) @loiclec * Significantly improve the performance of the `exactness` ranking rule for search requests containing many words (#3269 and meilisearch/milli#709) @loiclec * Translate multi-word synonyms into [phrases](https://docs.meilisearch.com/learn/what_is_meilisearch/features.html#phrase-search) when interpreting a search query. This change improves the relevancy of the results and stabilises the latency of search queries containing many multi-word synonyms, removing a source of denial-of-service attacks. (#3269 and meilisearch/milli#732) @loiclec * Improve the performance of the `proximity` ranking rule for search requests ending with short words (#3269 and meilisearch/milli#733) @loiclec * Limit the reindexing caused by updating settings when not needed (meilisearch/milli#706 and #3269) @GregoryConrad * Reduce incremental indexing time for the proximity ranking rule (#3445 and meilisearch/milli#776) @loiclec * Improve soft-deletion computation (meilisearch/milli#747) @dureuill ## Other improvements * Using a single dump, you can now migrate from any old version of Meilisearch (supporting the dump feature) to the latest version. We hope to facilitate your migration to v1.0.0 to make it as smooth as possible (#3170) @dureuill * Improve [`download-latest.sh`](https://github.com/meilisearch/meilisearch/blob/main/download-latest.sh) script: integrate apple-silicon binary (#3169) @curquiza * Clarify error message when the DB and engine versions are incompatible (#3279) @dureuill # Breaking changes ⚠️ Since we are releasing v1.0.0, this is the last time we introduce breaking changes until v2.0.0. ## New CLI behavior * As a security measure, Meilisearch will now reject master keys that are less than 16 bytes in [production environment](https://docs.meilisearch.com/learn/configuration/instance_options.html#environment) (#3274 and #3295) @dureuill * Remove `--max-index-size` and `--max-task-db` configuration options (#3278) @dureuill These options were not serving their intended purpose of limiting the disk space taken by Meilisearch for the reasons laid out [here](https://github.com/meilisearch/product/discussions/567#discussioncomment-4322126). Please [reach out](https://github.com/meilisearch/product/discussions/567#discussioncomment-4634432) if the new limits (described in the previous link) impede your usage. * Remove `--disable-auto-batching` CLI option and the associated environment variable `MEILI_DISABLE_AUTO_BATCHING` (#3296) @loiclec This option was initially introduced as a workaround in case of a bug and is no longer useful since the feature has matured. * Rename dump command from `--dumps-dir` to `--dump-dir` (#3175) @dureuill * Remove `--snapshot-interval-sec`. `--schedule-snapshot` now accepts an optional integer value specifying the interval in seconds (#3281) @dureuill * The `--log-level` option and `MEILI_LOG_LEVEL` environment variable now only accept the following arguments: `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE` and `OFF` (#3293) @loiclec * Remove hidden but usable CLI arguments `--nb-max-chunks` and `--log-every-n` (#3305) @Kerollmops ## Revamp primary key inference We have improved the primary key inference process when receiving documents for the first time. If an index's primary key has not been specified, Meilisearch will attempt to infer the primary key by searching for any attributes ending with `id`, such as `puid` or` _id`. If only one such attribute is detected, it will be designated as the index's primary key. If multiple attributes ending with `id` are detected, Meilisearch will ask you to specify the primary key instead of choosing the first found one. If no primary key candidates are detected, Meilisearch will throw an error as before. Done in meilisearch/milli#752 and #3269 by @dureuill ## Change the name of the binary when using `apt` The command used to install the Meilisearch binary with`apt` is now `apt install meilisearch` and not `apt install meilisearch-http`. To download the Meilisearch binary released before v1.0, you still need to use `apt install meilisearch-http`. ## Error handler changes All detailed task responses contain the `error` field. When a task fails, it is always accompanied by a JSON-formatted error response that contains an error `code` and `type`. Many of these `code`s and `type`s were unclear and have been changed to improve the user experience. This is a breaking change if you use any of the following `code`s or `type`s in your code base: * When calling `POST /indexes/swap-indexes`, the following error `code`s have been updated: <details> <summary>Click for details</summary> <ul> <li><code>duplicate_index_found</code> is renamed to <code>invalid_swap_duplicate_index_found</code></li> <li>Sending an array of indexes not containing exactly 2 <code>indexUids</code> for a swap operation object now returns <code>invalid_swap_indexes</code> instead of a <code>bad_request</code></li> <li>If the <code>indexes</code> field is missing, Meilisearch now returns <code>missing_swap_indexes</code> instead of <code>missing_parameter</code></li> </ul> </details> * When calling `GET /indexes` and `GET /indexes/:uid`, the following error `code`s have been updated: <details> <summary>Click for details</summary> <ul> <li>Sending an invalid <code>limit</code> parameter now returns <code>invalid_index_limit</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>offset</code> parameter now returns <code>invalid_index_offset</code> instead of <code>bad_request</code></li> </ul> </details> * When calling `POST /indexes` and `PUT /indexes/:uid`, the following error `code`s have been updated: <details> <summary>Click for details</summary> <ul> <li>A missing <code>uid</code> in the payload will return <code>missing_index_uid</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>primaryKey</code> now returns <code>invalid_index_primary_key</code> instead of <code>bad_request</code></li> </ul> </details> * When calling `GET /index/:uid/documents` and `GET /indexes/:uid/documents/:uid`, the following error `code`s have been updated:. <details> <summary>Click for details</summary> <ul> <li>Sending an invalid<code>fields</code> parameter now returns <code>invalid_document_fields</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>limit</code> parameter now returns <code>invalid_document_limit</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>offset</code> parameter now returns <code>invalid_document_offset</code> instead of <code>bad_request</code></li> <li>When adding documents, trying to update the primary-key now throw an error</li> </ul> </details> * When calling `POST /index/:uid/documents` and `PUT /indexes/:uid/documents` <details> <summary>Click for details</summary> <ul> <li>Sending an invalid <code>primaryKey</code> parameter now returns <code>invalid_index_primary_key</code> instead of <code>bad_request</code></li> <li><code>invalid_document_geo_field</code> is renamed to <code>invalid_geo_field</code></li> </ul> </details> * When calling `GET /index/:uid/search` and `POST /indexes/:uid/search`, the following error `code`s have been updated: <details> <summary>Click for details</summary> <ul> <li>Sending an invalid <code>q</code> parameter now returns <code>invalid_search_q</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>offset</code> parameter now returns <code>invalid_search_offset</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>limit</code> parameter now returns <code>invalid_search_limit</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>page</code> parameter now returns <code>invalid_search_page</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>hits_per_page</code> parameter now returns <code>invalid_search_hits_per_page</code> instead of <code>bad_request</code></li> <li>Sending an invalid<code>attributesToRetrieve</code> parameter now returns <code>invalid_search_attributes_to_retrieve</code> instead of <code>bad_request</code></li> <li>Sending an invalid<code>attributesToCrop</code> parameter now returns <code>invalid_search_attributes_to_crop</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>showMatchesPosition</code> parameter now returns <code>invalid_search_show_matches_position</code> instead of <code>bad_request</code></li> <li>Sending an invalid<code>filter</code> parameter now returns <code>invalid_search_filter</code> instead of <code>invalid_filter</code></li> <li>Sending an invalid <code>sort</code> parameter now returns <code>invalid_search_sort</code> instead of <code>invalid_sort</code></li> <li>Sending an invalid<code>facets</code> parameter now returns <code>invalid_search_facets</code> instead of <code>bad_request</code></li> <li>Sending an invalid<code>highlightPreTag</code> parameter now returns <code>invalid_search_highlight_pre_tag</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>highlightPostTag</code> parameter now returns <code>invalid_search_highlight_post_tag</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>matchingStrategy</code> parameter now returns <code>invalid_search_matching_strategy</code> instead of <code>bad_request</code></li> </ul> </details> * When calling any `/settings` routes and their sub-routes, the following error `code`s have been updated: <details> <summary>Click for details</summary> <ul> <li>Sending an invalid value for the <code>displayedAttributes</code> setting now returns <code>invalid_settings_displayed_attributes</code> instead of <code>bad_request</code></li> <li>Sending an invalid value for the <code>searchableAttributes</code> setting now returns <code>invalid_settings_searchable_attributes</code> instead of <code>bad_request</code></li> <li>Sending an invalid value for the <code>filterableAttributes</code> setting now returns <code>invalid_settings_filterable_attributes</code> instead of <code>bad_request</code></li> <li>Sending an invalid value for the <code>sortableAttributes</code> setting now returns <code>invalid_settings_sortable_attributes</code> instead of <code>bad_request</code></li> <li>Sending an invalid value for the <code>rankingRules</code> setting now returns <code>invalid_settings_ranking_rules</code> instead of <code>bad_request</code></li> <li>Sending an invalid value for the <code>stopWords</code> setting now returns <code>invalid_settings_stop_words</code> instead of <code>bad_request</code></li> <li>Sending an invalid value for the <code>synonyms</code> setting now returns <code>invalid_settings_synonyms</code> instead of <code>bad_request</code></li> <li>Sending an invalid value for the <code>distinctAttribute</code> setting now returns <code>invalid_settings_distinct_attribute</code> instead of <code>bad_request</code></li> <li>Sending an invalid value for the <code>typoTolerance</code> setting now returns <code>invalid_settings_typo_tolerance</code> instead of <code>bad_request</code></li> <li><code>invalid_settings_typo_tolerance</code> replaces <code>invalid_typo_tolerance_min_word_size_for_typos</code></li> <li>Sending an invalid value for the <code>faceting</code> setting now returns <code>invalid_settings_faceting</code> instead of <code>bad_request</code></li> <li>Sending an invalid value for the <code>pagination</code> setting now returns <code>invalid_settings_pagination</code> instead of <code>bad_request</code></li> </ul> </details> * When calling any `/tasks` routes, the following error `code`s have been updated: <details> <summary>Click for details</summary> <ul> <li>Sending an invalid <code>offset</code> parameter now returns <code>invalid_search_offset</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>limit</code> parameter now returns <code>invalid_search_limit</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>uids</code> parameter now returns <code>invalid_task_uids</code> instead of <code>invalid_task_uids_filter</code></li> <li>Sending an invalid <code>types</code> parameter now returns <code>invalid_task_types</code> instead of <code>invalid_task_types_filter</code></li> <li>Sending an invalid <code>statuses</code> parameter now returns <code>invalid_task_statuses</code> instead of <code>invalid_task_statuses_filter</code></li> <li>Sending an invalid <code>cancelBy</code> parameter now returns <code>invalid_task_cancel_by</code> instead of <code>invalid_task_canceled_by_filter</code></li> <li>Sending an invalid <code>beforeEnqueuedAt</code> parameter now returns <code>invalid_task_before_enqueued_at</code> instead of <code>invalid_task_date_filter</code></li> <li>Sending an invalid <code>afterEnqueuedAt</code> parameter now returns <code>invalid_task_after_enqueued_at</code> instead of <code>invalid_task_date_filter</code></li> <li>Sending an invalid <code>beforeStartedAt</code> parameter now returns <code>invalid_task_before_started_at</code> instead of <code>invalid_task_date_filter</code></li> <li>Sending an invalid <code>afterStartedAt</code> parameter now returns <code>invalid_task_after_started_at</code> instead of <code>invalid_task_date_filter</code></li> <li>Sending an invalid <code>beforeFinishedAt</code> parameter now returns <code>invalid_task_before_finished_at</code> instead of <code>invalid_task_date_filter</code></li> <li>Sending an invalid <code>afterFinishedAt</code> parameter now returns <code>invalid_task_after_finished_at</code> instead of <code>invalid_task_date_filter</code></li> </ul> </details> * When calling `GET /keys` and `GET /keys/:uid`, the following error `code`s have been updated: <details> <summary>Click for details</summary> <ul> <li>Sending an invalid <code>limit</code> parameter now returns <code>invalid_api_key_limit</code> instead of <code>bad_request</code></li> <li>Sending an invalid <code>offset</code> parameter now returns <code>invalid_api_key_offset</code> instead of <code>bad_request</code></li> </ul> </details> * When calling `POST /keys`, the following error `code`s have been updated: <details> <summary>Click for details</summary> <ul> <li>If the <code>actions</code> field is missing, Meilisearch now returns <code>missing_api_key_actions</code> instead of <code>missing_parameter</code></li> <li>If the <code>indexes</code> field is missing, Meilisearch now returns <code>missing_api_key_indexes</code> instead of <code>missing_parameter</code></li> <li>If the <code>expiresAt</code> field is missing, Meilisearch now returns <code>missing_api_key_expire_at</code> instead of <code>missing_parameter</code></li> </ul> </details> * When calling `PATCH /keys`, the following error `code`s have been updated: <details> <summary>Click for details</summary> <ul> <li>Updating the <code>uid</code> field now returns <code>immutable_api_key_uid</code> instead of <code>immutable_field</code></li> <li>Updating the <code>actions</code> field now returns <code>immutable_api_key_actions</code> instead of <code>immutable_field</code></li> <li>Updating the <code>indexes</code> field now returns <code>immutable_api_key_indexes</code> instead of <code>immutable_field</code></li> <li>Updating the <code>expiresAt</code> field now returns <code>immutable_api_key_expires_at</code> instead of <code>immutable_field</code></li> <li>Updating the <code>createdAt</code> field now returns <code>immutable_api_key_created_at</code> instead of <code>immutable_field</code></li> <li>Updating the <code>updatedAt</code> field now returns <code>immutable_api_key_updated_at</code> instead of <code>immutable_field</code></li> </ul> </details> * When Meilisearch does not have enough space - the `no_space_left_on_device` code replaces `internal` - `system` type replaces `internal` * When Meilisearch throws an I/O error - the `io_error` code replaces `internal` - `system` type replaces `internal` * When your machines exceeds the maximum number allowed for open files, Meilisearch returns - the `too_many_open_files` code instead of `internal` - `system` type instead of `internal` * All Meilisearch errors of type `system` return the HTTP error code `500` (#3321) @irevoire * The following errors are now returned synchronously: - `invalid_index_uid` - `invalid_settings_ranking_rules` - `invalid_settings_typo_tolerance` if `oneTypo` and `twoTypos` are filled (for `minWordSizeForTypos`) but are invalid Done by @irevoire and @loiclec in #3316, #3263, #3329, #3334, #3412 and #3339 # Fixes 🐞 * Commit SHA and date are now present in the Meilisearch logs when launching Meilisearch from a Docker image (#3212) @brunoocasali * Fix `MDB_PAGE_FULL` internal error that would sometimes occur when inserting documents on macOS (#3210) @Kerollmops * Re-introduce the `no_space_left_on_device` error (#3263) @irevoire * Index creation and update dates are now kept when using a dump (#3245) @amab8901 * Fix the usage of `*` for the `tasks` routes. More information on this issue (#3339) @loiclec * Fix `/tasks/cancel` behavior when sending an invalid `canceledBy` parameter (#3375) @Kerollmops * Fix dumps import for Windows, see [this issue](https://github.com/meilisearch/meilisearch/issues/2878) for more information. * Return an error when a `_geo` field contains one extra unspecified field (#3415 meilisearch/milli#772) @irevoire * Fix error message when `lat` and `lng` are unparseable (#3415 meilisearch/milli#763) @ahlner * Fix the `databaseSize` value of the `/stats` route (#3418) @irevoire # Misc The following changes do not impact user experience. * GitHub CIs * Replace deprecated set-output in GitHub actions (#3015) @funilrys * Use ubuntu-latest when not impacting (#3183) @curquiza * Add CI to push a `latest` git tag for every stable Meilisearch release (#3172 and #3207) @curquiza * Add `workflow_dispatch` to flaky.yml (#3041) @curquiza * Add a dispatch to the publish binaries workflow (#3100) @curquiza * Remove macos-latest and windows-latest usages (#3264) @curquiza * Add a nightly CI: create a `nightly` Docker tag every day based on the latest commit on `main` (#3229 and #3247) @curquiza * Use ubuntu-18.04 container instead of GitHub hosted actions (#3261) @curquiza * Test suite * Re-enable dump tests (#3119) @irevoire * Re-enable dump test on dates (#3188) @irevoire * Stop snapshotting the version of Meilisearch in the dump test (#3208) @irevoire * Update insta snap tests for index dates of dump v5 (#3302) @loiclec * Add a test on finite pagination placeholder search when distinct is enabled (#3251) @ManyTheFish * Dependencies * Fix update-cargo-toml-version.yml (#3224) @mohitsaxenaknoldus * Update CI dependencies (#3179, #3181 and #3180) * Update some `cargo_toml` dependency (#3128) @colbsmcdolbs * Update mini-dashboard to v0.2.5 (#3186, and #3322) @mdubus * Documentation * Update README to reference Meilisearch Cloud (#3221) @davelarkan * Replace Slack link with Discord in README.md (#3234 and #3276) @tpayet @shivaylamba * Internal changes * Implement Uuid codec for heed (#3001) @elbertronnie * Improve how we receive the documents payload (#3164 and #3266) @jiangbo212 * Improve clarity of the code that receives payloads (#3236) @Kerollmops * Extract the dates out of the dumpv4 (#3012 and #3190) @funilrys @irevoire * Rename meilisearch-http (#3112) @colbsmcdolbs ❤️ Thanks again to our external contributors: - [Meilisearch](https://github.com/meilisearch/meilisearch): @amab8901, @colbsmcdolbs, @elbertronnie, @funilrys, @jiangbo212, @mohitsaxenaknoldus, and @shivaylamba. - [Milli](https://github.com/meilisearch/milli): @amab8901, @GregoryConrad and @pnhatminh. - [Charabia](https://github.com/meilisearch/charabia): @choznerol, @crudiedo, @daniel-shuy, @harshalkhachane, @mosuka, @qbx2, @Roms1383, @Sokom141, and @yenwel.