Meilisearch: v1.42.0 🦑​

# **✨ Enhancement** ## Support search fallback on remote unavailability By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6306 We introduce a new fallback system for the sharding and replication enterprise edition feature, along with a way to determine which remote is available. The engine can avoid machines that are unavailable for a period and resume querying them once they're back online. The following snippet shows what the /network route looks like now that this PR exposes the remote statuses/availabilities. ```json "remotes": { "prod2": { "url": "http://localhost:7702", "searchApiKey": "mykey", "writeApiKey": "mykey", "status": "available" }, "prod3": { "url": "http://localhost:7703", "searchApiKey": "mykey", "writeApiKey": "mykey", "status": "unavailable" } } ``` ## 🔬 Experimental: Document join Filtering By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6314 This enhancement extends the `Cross-index document hydration introduced` in [v1.39.0](https://github.com/meilisearch/meilisearch/releases/tag/v1.39.0) by allowing the user to filter on the foreign indexes to retrieve the documents. > 📓 Note: This implementation doesn't support a remote sharding environment ### `foreignKeys` experimental feature The`foreignKeys` experimental feature must be activated to use the foreign filters: ``` curl -X PATCH 'http://127.0.0.1:7700/experimental-features' \ -H 'Content-Type: application/json' \ --data-binary '{"foreignKeys": true}' ``` ### `foreignKeys` + `filter` index setting To be able to use the foreign filters, the related field must be set as a foreignKey and as a filterableAttribute in `/indexes/{index_uid}/settings`: ```jsonc { // new setting, an array of foreign keys that allows multiple foreign relationships between indexes "foreignKeys": [ { // the path in the JSON document containing foreign document ids "fieldName": "actors", // the UID of the foreign index containing the documents to fetch during hydration "foreignIndexUid": "actors" } ], // the actors field must be filterable on equality "filterableAttributes": [ { "attributePatterns": ["actors"], "features": { "facetSearch": false, "filter": { "equality": true, "comparison": false } } } ] } ``` ### filtering using the `_foreign` filter On the search route, a new `_foreign` verb has been introduced and should be used as follows: ```jsonc { "q": "<query>", // filters on the movie index: // genres = action // AND // the foreign documents from the actor index match: birthday STARTS WITH \"1958-\" AND popularity >= 3.5 "filter": "genres = action AND _foreign(actors, birthday STARTS WITH \"1958-\" AND popularity >= 3.5)" } ``` > Note: nesting foreign filters is not supported and will return an error <details> <summary> ### Example of usage </summary> #### Prerequisites * Meilisearch running on `127.0.0.1:7700` on the `document-join-hydration` branch. #### Step 1: Enable Foreign Keys Feature ```bash curl -X PATCH 'http://127.0.0.1:7700/experimental-features' \ -H 'Content-Type: application/json' \ --data-binary '{"foreignKeys": true}' ``` #### Step 2: Create Indexes ##### Create the `actors` index ```bash curl -X POST 'http://127.0.0.1:7700/indexes' \ -H 'Content-Type: application/json' \ --data-binary '{"uid": "actors", "primaryKey": "id"}' ``` ##### Create the `movies` index ```bash curl -X POST 'http://127.0.0.1:7700/indexes' \ -H 'Content-Type: application/json' \ --data-binary '{"uid": "movies", "primaryKey": "id"}' ``` #### Step 3: Add Documents to the `actors` Index ```bash curl -X POST 'http://127.0.0.1:7700/indexes/actors/documents' \ -H 'Content-Type: application/json' \ --data-binary '[ {"id": 1, "name": "Tom", "familyName": "Hanks", "birthDate": "1956-07-09"}, {"id": 2, "name": "Meryl", "familyName": "Streep", "birthDate": "1949-06-22"}, {"id": 3, "name": "Leonardo", "familyName": "DiCaprio", "birthDate": "1974-11-11"}, {"id": 4, "name": "Emma", "familyName": "Watson", "birthDate": "1990-04-15"} ]' ``` #### Step 4: Add Documents to the `movies` Index ```bash curl -X POST 'http://127.0.0.1:7700/indexes/movies/documents' \ -H 'Content-Type: application/json' \ --data-binary '[ {"id": 1, "title": "Forrest Gump", "description": "The presidencies of Kennedy and Johnson, the Vietnam War, the Watergate scandal and other historical events unfold from the perspective of an Alabama man with an IQ of 75.", "actors": [1]}, {"id": 2, "title": "The Devil Wears Prada", "description": "A smart but sensible new graduate lands a job as an assistant to Miranda Priestly, the demanding editor-in-chief of a high fashion magazine.", "actors": [2, 4]}, {"id": 3, "title": "Inception", "description": "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.", "actors": [3]}, {"id": 4, "title": "Cast Away", "description": "A FedEx executive undergoes a physical and emotional transformation after crash landing on a deserted island.", "actors": [1]} ]' ``` #### Step 5: Configure Foreign Keys on the `movies` Index ```bash curl -X PATCH 'http://127.0.0.1:7700/indexes/movies/settings' \ -H 'Content-Type: application/json' \ --data-binary '{"foreignKeys": [{"fieldName": "actors", "foreignIndexUid": "actors"}], "filterableAttributes": [{"attributePatterns": ["actors"],"features": {"facetSearch": false,"filter": {"equality": true,"comparison": false}}}]}' ``` #### Step 6: Configure filterable on the `actors` Index ```bash curl -X PATCH 'http://127.0.0.1:7700/indexes/actors/settings' \ -H 'Content-Type: application/json' \ --data-binary '{"filterableAttributes": [{"attributePatterns": ["birthDate"],"features": {"facetSearch": false,"filter": {"equality": true,"comparison": false}}}]}' ``` #### Step 7: Perform a Federated Search ```bash curl -X POST 'http://127.0.0.1:7700/multi-search' \ -H 'Content-Type: application/json' \ --data-binary '{ "queries": [ { "indexUid": "movies", "q": "Forrest", "filter": "_foreign(actors, birthDate = \"1956-07-09\")" } ], "federation": { "limit": 20, "offset": 0 } }' ``` #### Expected Result The federated search should return movie documents with the `actors` array automatically hydrated with full actor objects instead of just IDs: ```json { "hits": [ { "id": 1, "title": "Forrest Gump", "description": "...", "actors": [ { "id": 1, "name": "Tom", "familyName": "Hanks", "birthDate": "1956-07-09" } ], "_federation": { "indexUid": "movies", "queriesPosition": 0, "weightedRankingScore": 0.9848484848484849 } } ], "processingTimeMs": 208, "limit": 20, "offset": 0, "estimatedTotalHits": 1 } ``` </details> # 🪲 Bug fixes - Fix a race condition when writing network by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6300 We fixed a race condition in network topology changes that could cause errors and prevent documents from being correctly indexed. Additionally, we fixed a bug in the `networkTopologyChange` task batching that was causing it to batch too many task types. We made sure it only batches import tasks, and only those, to avoid out-of-order task processing. - Throw document template errors when updating the chat settings by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6321 We fixed an issue that prevented the engine from explicitly showing the possible document template errors users could encounter when updating the template in the chat settings. The engine now correctly checks for and throws template errors when they are detected. - Fix: Update Index tasks will be properly forwarded to remote nodes by @dureuill in https://github.com/meilisearch/meilisearch/pull/6299 - Fix action mistake on the chat completions route by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6290 # **🔩 Miscellaneous** - Use the latest version of heed with nested rtxns support by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6316 This PR bumps the versions of crates that use heed to [the latest version, v0.22.1](https://github.com/meilisearch/heed/releases/tag/v0.22.1). This version finally stabilized a long-standing piece of work we were doing with Howard Chu: nested read transactions. We no longer have to rely on unstable pre-releases, but rather on a clean, stable version of LMDB (still a fork, but a better one). - Add section to [CONTRIBUTING.md](http://contributing.md/) to bump mini-dashboard version and testing section to right place by @curquiza in https://github.com/meilisearch/meilisearch/pull/6195 - Make the no-agent [AGENTS.ms](http://agents.ms/) more permissive by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6260 - Remove deleted test commands by @Strift in https://github.com/meilisearch/meilisearch/pull/6283 - Fix OpenAPI schema generation for chat completions route by @qdequele in https://github.com/meilisearch/meilisearch/pull/6274 - Rename OpenAPI route names for search rules and compact by @qdequele in https://github.com/meilisearch/meilisearch/pull/6298 - Update README with new features and demos by @qdequele in https://github.com/meilisearch/meilisearch/pull/6297 - Prevent shell injection in benchmark workflows by @curquiza & @Kerollmopsin https://github.com/meilisearch/meilisearch/pull/6308 & https://github.com/meilisearch/meilisearch/pull/6318 - Rename some of the search performance traces by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6323