Meilisearch: v1.7.0 🐇

Meilisearch v1.7.0 focuses on improving v1.6.0 features, indexing speed and hybrid search. <!-- The following lines should NOT be put in the PRE-release changelogs --> 🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment happens between 4 to 48 hours after a new version becomes available. <!-- The following lines should NOT be put in the PRE-release changelogs --> Some SDKs might not include all new features—consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️). # New features and improvements 🔥 ## Improved AI-powered search — Experimental To activate AI-powered search, set `vectorStore` to `true` in [the `/experimental-features` route](https://www.meilisearch.com/docs/reference/api/experimental_features). Consult the [Meilisearch documentation](https://www.meilisearch.com/docs/learn/experimental/vector_search#vector-search-with-auto-embeddings) for more information. 🗣️ This is an experimental feature and we need your help to improve it! Share your thoughts and feedback on this [GitHub discussion](https://github.com/orgs/meilisearch/discussions/677). ### New OpenAI embedding models When [configuring OpenAI embedders](https://www.meilisearch.com/docs/learn/experimental/vector_search#generate-auto-embeddings-with-openai)), you can now specify two new models: - `text-embedding-3-small` with a default dimension of 1536. - `text-embedding-3-large` with a default dimension of 3072. These new models are cheaper and improve search result relevancy. ### Custom OpenAI model dimensions You can configure `dimensions` for sources using the new OpenAI models: `text-embedding-3-small` and `text-embedding-3-large`. Dimensions must be bigger than 0 and smaller than the model size: ```json5 "embedders": { "new_model": { "source": "openAi", "model": "text-embedding-3-large", "dimensions": 512 // must be >0, must be <= 3072 for "text-embedding-3-large" }, "legacy_model": { "source": "openAi", "model": "text-embedding-ada-002" } } ``` You cannot customize dimensions for older OpenAI models such as `text-embedding-ada-002`. Setting `dimensions` to any value except the default size of these models will result in an error. Done in #4375 by @Gosti. ### GPU support when computing Hugging Face embeddings Activate CUDA to use Nvidia GPUs when computing Hugging Face embeddings. This can significantly improve embedding generation speeds. To enable GPU support through CUDA for HuggingFace embedding generation: 1. [Install CUDA dependencies](https://huggingface.github.io/candle/guide/installation.html) 2. Clone and compile Meilisearch with the `cuda` feature: `cargo build --release --package meilisearch --features cuda` 3. Launch your freshly compiled Meilisearch binary 4. [Activate vector search](https://www.meilisearch.com/docs/learn/experimental/vector_search#activate-vector-search) 5. [Add a Hugging Face embedder](https://www.meilisearch.com/docs/learn/experimental/vector_search#generate-auto-embeddings-with-huggingface) Done by @dureuill in #4304. ## Improved indexing speed and reduced memory crashes * Auto-batch task deletion to reduce indexing time (#4316) @irevoire * Improved indexing speed for vector store ([Hybrid search experimental feature](https://github.com/orgs/meilisearch/discussions/677) indexing time more than 10 times faster) (#4332) @Kerollmops @irevoire * Capped the maximum memory of grenad sorters to reduce memory usage (#4388) @Kerollmops * Added [multiple technical and internal indexing improvements](https://github.com/meilisearch/meilisearch/pull/4350) (#4350) @ManyTheFish * Enhance facet incremental indexing (#4433) @ManyTheFish * Change the threshold triggering incremental indexing (#4462) @ManyTheFish ## Stabilized `showRankingScoreDetails` The `showRankingScoreDetails` search parameter, first introduce as an experimental feature in Meilisearch [v1.3.0](https://github.com/meilisearch/meilisearch/releases/tag/v1.3.0), is now a stable feature. Use it with the `/search` endpoint to view detailed scores per ranking rule for each returned document: ```bash curl \ -X POST 'http://localhost:7700/indexes/movies/search' \ -H 'Content-Type: application/json' \ --data-binary '{ "q": "Batman Returns", "showRankingScoreDetails": true }' ``` When `showRankingScoreDetails` is set to `true`, returned documents include a `_rankingScoreDetails` field: ```json "_rankingScoreDetails": { "words": { "order": 0, "matchingWords": 1, "maxMatchingWords": 1, "score": 1.0 }, "typo": { "order": 1, "typoCount": 0, "maxTypoCount": 1, "score": 1.0 }, "proximity": { "order": 2, "score": 1.0 }, "attribute": { "order": 3, "attributes_ranking_order": 0.8, "attributes_query_word_order": 0.6363636363636364, "score": 0.7272727272727273 }, "exactness": { "order": 4, "matchType": "noExactMatch", "matchingWords": 0, "maxMatchingWords": 1, "score": 0.3333333333333333 } } ``` Done by @dureuill in #4389. ## Improved logging Done by @irevoire in #4391 ### Log output modified Log messages now follow a different pattern: ```bash # new format ✅ 2024-02-06T14:54:11Z INFO actix_server::builder: 200: starting 10 workers # old format ❌ [2024-02-06T14:54:11Z INFO actix_server::builder] starting 10 workers ``` ⚠️ This change may impact you if you have any automated tasks based on log output. ### Log output format — Experimental You can now configure Meilisearch to output logs in JSON. Relaunch your instance passing `json` to the `--experimental-logs-mode` command-line option: ```bash ./meilisearch --experimental-logs-mode json ``` `--experimental-logs-format` accepts two values: - `human`: default human-readable output - `json`: JSON structured logs 🗣️ This feature is experimental and we need your help to improve it! Share your thoughts and feedback on this [GitHub discussion](https://github.com/orgs/meilisearch/discussions/723). ⚠️ Experimental features may be incompatible between Meilisearch versions. ### New `/logs/stream` and `/logs/stderr` routes — Experimental Meilisearch v1.7 introduces 2 new experimental API routes: `/logs/stream` and `/logs/stderr`. Use the `/experimental-features` route to activate both routes during runtime: ```bash curl \ -X PATCH 'http://localhost:7700/experimental-features/' \ -H 'Content-Type: application/json' \ --data-binary '{ "logsRoute": true }' ``` 🗣️ This feature is experimental, and we need your help to improve it! Share your thoughts and feedback on this [GitHub discussion](https://github.com/orgs/meilisearch/discussions/721). ⚠️ Experimental features may be incompatible between Meilisearch versions. #### `/logs/stream` Use the `POST` endpoint to output logs in a stream. The following example disables actix logging and keeps all other logs at the `DEBUG` level: ```bash curl \ -X POST http://localhost:7700/logs/stream \ -H 'Content-Type: application/json' \ --data-binary '{ "mode": "human", "target": "actix=off,debug" }' ``` This endpoint requires two paramaters: - `target`: defines the log level and on which part of the engine you want to apply it. Must be a string formatted as `code_part=log_level`. Omit `code_part=` to set a single log level for the whole strram. Valid values for log level are: `trace,` `debug,` `info,` `warn,` `error`, or `off` - `mode`: accepts `fmt` (basic) or `profile` (verbose trace) Use the `DELETE` endpoint of `/logs/stream` to interrupt a stream: ```bash curl -X DELETE http://localhost:7700/logs/stream ``` You may only have one listener at a time. Meilisearch log streams are not compatible with `xh` or `httpie`. #### `/logs/stderr` Use the `POST` endpoint to configure the default log output for non-stream logs: ```bash curl \ -X POST http://localhost:7700/logs/stream \ -H 'Content-Type: application/json' \ --data-binary '{ "target": "debug" }' ``` `/logs/stderr` accepts one parameter: - `target`: defines the log level and on which part of the engine you want to apply it. Must be a string formatted as `code_part=log_level`. Omit `code_part=` to set a single log level for the whole strram. Valid values for log level are: `trace,` `debug,` `info,` `warn,` `error`, or `off` ## Other improvements * [Prometheus experimental feature](https://github.com/orgs/meilisearch/discussions/625): add job variable to Grafana dashboard (#4330) @capJavert * Multiple language support improvements, including expanded Vietnamese normalization (Ð and Đ into d). Now uses Charabia [v0.8.7](https://github.com/meilisearch/charabia/releases/tag/v0.8.7). (#4365) @agourlay, @choznerol, @ngdbao, @timvisee, @xshadowlegendx, and @ManyTheFish * New experimental feature: [change the behavior of Meilisearch in a few ways to run meilisearch in a cluster by externalizing the task queue](https://github.com/orgs/meilisearch/discussions/725). * Add the content type to the webhook (#4450) @irevoire # Fixes 🐞 * Make update file deletion atomic (#4435) @irevoire * Do not omit vectors when importing a dump (#4446) @dureuill * Put a bound on OpenAI timeout (#4459) @dureuill # Misc * Dependencies upgrade * Bump rustls-webpki from 0.101.3 to 0.101.7 (#4263) * Bump h2 from 0.3.20 to 0.3.24 (#4345) * Update the dependencies (#4332) @Kerollmops * CIs and tests * Update SDK test dependencies (#4293) @curquiza * Remove tests on nightly (#4353) @dureuill * Add subcommand to run benchmarks (#4445) @dureuill * Documentation * Add Setting API reminder in issue template (#4325) @ManyTheFish * Update README (#4319) @codesmith-emmy * Fix broken links in README.md (#4360) @Elliot67 * Misc * Fix compilation warnings (#4295) @irevoire ❤️ Thanks again to our external contributors: - [Meilisearch](https://github.com/meilisearch/meilisearch): @capJavert, @codesmith-emmy, @Elliot67 and @Gosti - [Charabia](https://github.com/meilisearch/charabia): @agourlay, @choznerol, @ngdbao, @timvisee, and @xshadowlegendx