diff --git a/open-api.yaml b/open-api.yaml index ca9a4ac4..26bea94e 100644 --- a/open-api.yaml +++ b/open-api.yaml @@ -2,7 +2,7 @@ openapi: 3.1.0 info: title: Meilisearch Core API description: 'Search documents, configure and manage the Meilisearch engine.' - version: 1.5.0 + version: 1.6.0 contact: name: Meilisearch email: bonjour@Meilisearch.com diff --git a/text/0034-telemetry-policies.md b/text/0034-telemetry-policies.md index 4431d954..d117e02d 100644 --- a/text/0034-telemetry-policies.md +++ b/text/0034-telemetry-policies.md @@ -59,6 +59,7 @@ The collected data is sent to [Segment](https://segment.com/). Segment is a plat | Pagination Updated | Occurs when pagination settings are updated via `PATCH` — `/indexes/:indexUid/settings/pagination`. | | Faceting Updated | Occurs when faceting settings are updated via `PATCH` — `/indexes/:indexUid/settings/faceting`. | | DistinctAttribute Updated | Occurs when distinct attribute setting is updated via `PUT` — `/indexes/:indexUid/settings/distinct-attribute`. | +| ProximityPrecision Updated | Occurs when proximity precision setting is updated via `PUT` — `/indexes/:indexUid/settings/proximity-precision`. | | DisplayedAttributes Updated | Occurs when displayed attributes are updated via `PUT` — `/indexes/:indexUid/settings/displayed-attributes`. | | StopWords Updated | Occurs when stop words are updated via `PUT` — `/indexes/:indexUid/settings/stop-words`. | | SeparatorTokens Updated | Occurs when separator tokens are updated via `PUT` — `/indexes/:indexUid/settings/separator-tokens`. | @@ -107,6 +108,7 @@ The collected data is sent to [Segment](https://segment.com/). Segment is a plat | `infos.ssl_resumption` | `true` if `--ssl-resumption`/`MEILI_SSL_RESUMPTION` is specified, otherwise `false` | false | Every Hour | | `infos.ssl_tickets` | `true` if `--ssl-tickets`/`MEILI_SSL_TICKETS` is specified, otherwise `false` | false | Every Hour | | `infos.with_configuration_file` | `true` if the instance is launched with a configuration file, otherwise `false` | false | Every Hour | +| `infos.task_queue_webhook` | `true` if the instance is launched with a task queue webhook, otherwise `false` | false | Every Hour | | `infos.experimental_enable_metrics` | `true` if `--experimental-enable-metrics`/`MEILI_EXPERIMENTAL_ENABLE_METRICS` is specified at launch, otherwise `false` | `false` | Every Hour | | `infos.experimental_reduce_indexing_memory_usage` | `true` if `--experimental-reduce-indexing-memory-usage`/`MEILI_EXPERIMENTAL_REDUCE_INDEXING_MEMORY_USAGE` is specified at launch, otherwise `false` | `false` | Every Hour | | `system.distribution` | Distribution on which MeiliSearch is launched | Arch Linux | Every hour | @@ -169,6 +171,8 @@ The collected data is sent to [Segment](https://segment.com/). Segment is a plat | `faceting.sort_facet_values_by_star_count` | Whether the user set all fields to be sort by count | `true` | `Settings Updated`, `Faceting Updated` | | `faceting.sort_facet_values_by_total` | The number of different values that were set | `10` | `Settings Updated`, `Faceting Updated` | | `distinct_attribute.set` | `true` if a field name is specified as a distrinct attribute, otherwise `false`. | `false` | `Settings Updated`, `DistinctAttribute Updated` | +| `proximity_precision.set` | `true` the setting has been manually set, otherwise `false`. | `false` | `Settings Updated`, `ProximityPrecision Updated` | +| `proximity_precision.value` | `byWord` or `byAttribute` if the setting has been set, otherwise `null`. | `null` | `Settings Updated`, `ProximityPrecision Updated` | | `displayed_attributes.total` | Number of displayed attributes. | `3` | `SettingUpdated`, `DisplayedAttributes Updated` | | `displayed_attributes.with_wildcard` | `true` if `*` is specified as a displayed attribute, otherwise `false`. | `false` | `SettingUpdated`, `DisplayedAttributes Updated` | | `stop_words.total` | Number of stop words. | `3` | `Settings Updated`, `StopWords Updated` | @@ -254,6 +258,7 @@ This property allows us to gather essential information to better understand on | infos.ssl_resumption | `true` if `--ssl-resumption`/`MEILI_SSL_RESUMPTION` is specified, otherwise `false` | false | | infos.ssl_tickets | `true` if `--ssl-tickets`/`MEILI_SSL_TICKETS` is specified, otherwise `false` | false | | infos.with_configuration_file | `true` if the instance is launched with a configuration file, otherwise `false` | `false` | +| infos.task_queue_webhook | `true` if the instance is launched with a task queue webhook, otherwise `false` | `false` | | infos.experimental_enable_metrics | `true` if `--experimental-enable-metrics`/`MEILI_EXPERIMENTAL_ENABLE_METRICS` is specified at launch, otherwise `false` | `false` | ##### MeiliSearch Statistics `stats` @@ -495,6 +500,8 @@ This property allows us to gather essential information to better understand on | faceting.sort_facet_values_by_star_count | Whether the user set all fields to be sort by count | `true` | | faceting.sort_facet_values_by_total | The number of different values that were set | `10` | | distinct_attribute.set | `true` if a field name is specified, otherwise `false`. | `false` | +| proximity_precision.set | `true` the setting has been manually set, otherwise `false`. | `false` | +| proximity_precision.value | `byWord` or `byAttribute` if the setting has been set, otherwise `null`. | `null` | | displayed_attributes.total | Number of displayed attributes. | `3` | | displayed_attributes.with_wildcard | `true` if `*` is specified as a displayed attribute, otherwise `false`. | `false` | | stop_words.total | Number of stop words. | `3` | @@ -580,6 +587,14 @@ This property allows us to gather essential information to better understand on | user_agent | Represents the user-agent encountered on this call. | `["Meilisearch Ruby (v2.1)", "Ruby (3.0)"]` | | distinct_attribute.set | `true` if a field name is specified, otherwise `false`. | `false` | +## `ProximityPrecision Updated` + +| Property name | Description | Example | +|---------------|-------------|---------| +| user_agent | Represents the user-agent encountered on this call. | `["Meilisearch Ruby (v2.1)", "Ruby (3.0)"]` | +| proximity_precision.set | `true` the setting has been manually set, otherwise `false`. | `false` | +| proximity_precision.value | `byWord` or `byAttribute` if the setting has been set, otherwise `null`. | `byWord` | + ## `DisplayedAttributes Updated` | Property name | Description | Example | diff --git a/text/0119-instance-options.md b/text/0119-instance-options.md index 069c8fb7..f85baf13 100644 --- a/text/0119-instance-options.md +++ b/text/0119-instance-options.md @@ -119,6 +119,8 @@ The expected behavior of each flag is described in the list above. - [Config file path](#3326-config-file-path) - [Experimental enable Metrics](#3327-experimental-enable-metrics) - [Experimental reduce indexing memory usage](#3328-experimental-reduce-indexing-memory-usage) +- [Task webhook url](#3329-task-webhook-url) +- [Task webhook authorization header](#3330-task-webhook-authorization-header) #### 3.3.1. Database path @@ -498,6 +500,26 @@ Activate the `/metrics` endpoint to collect Meilisearch metrics for monitoring p Enables the `MDB_WRITEMAP` option of LMDB, making the internal key-value store use much less RAM than usual. +#### 3.3.29. Task webhook url + +**Environment variable**: `MEILI_TASK_WEBHOOK_URL` +**CLI option**: `--task-webhook-url` +**Default**: Disabled +**Expected value**: an url to your server + +Sets the URL where Meilisearch will send a notification every time it processes a batch of tasks. +⚠️ If the URL is not correctly formatted, it'll throw an error before starting. +⚠️ If the URL is correctly formatted but doesn't point to a valid server, Meilisearch won't return an error but will log one every time it processes a batch of tasks. + +#### 3.3.30. Task webhook authorization header + +**Environment variable**: `MEILI_TASK_WEBHOOK_AUTHORIZATION_HEADER` +**CLI option**: `--task-webhook-authorization-header` +**Default**: Disabled +**Expected value**: a string representing the value of the authorization header you want to send with the processed tasks + +Sets the authorization header value that meilisearch must send with the processed tasks. + ## 4. Technical Aspects N/A diff --git a/text/0123-proximity-precision.md b/text/0123-proximity-precision.md new file mode 100644 index 00000000..aa524d6d --- /dev/null +++ b/text/0123-proximity-precision.md @@ -0,0 +1,126 @@ +# Proximity Precision Setting API + +## 1. Summary + +This specification describes the `proximityPrecision` index setting API endpoints. + +## 2. Motivation +N/A + +## 3. Functional Specification + +### 3.1. Explanations + +Let the users the possibility to lose a piece of Meilisearch relevancy to gain a lot of indexing speed and disk space. +Changing this setting will impact the way Meilisearch calculate the proximity between the query words, +this changes the behavior of the `proximity` ranking-rule, the phrase search, and the multi-word synonyms. +There are 2 possible variants for this setting: + +#### 3.1.1. `byWord` + +This variant is the default variant when the setting is not set. + +Meilisearch will be precise when computing the proximity between words. It will calculate the distance as the minimal number of words in the document separating the queried words. +This variant forces Meilisearch to compute and create a dedicated database to store the distance between words which impacts the indexing time. + +#### 3.1.2. `byAttribute` + +Meilisearch will not be precise when computing the proximity between words. It will only calculate 2 degrees of distances,the queried words are considered close if they are in the same attribute, and will be considered far if there aren't. It will no more take into account the distance between words or their order in the query. + +### 3.2. Global Settings API Endpoints Definition + +`proximityPrecision` is a sub-resource of `/indexes/:index_uid/settings`. + +See [Settings API](0123-settings-api.md). + +### 3.3. API Endpoints Definition + +Manipulate the `proximityPrecision` setting of a Meilisearch index. + +#### 3.3.1. `GET` - `indexes/:index_uid/settings/proximity-precision` + +Fetch the `proximityPrecision` setting of a Meilisearch index. + +##### 3.3.1.1. Response Definition + +- Type: String +- Default: `byWord` + +##### 3.3.1.2. Errors + +- 🔴 Sending an invalid index uid format for the `:index_uid` path parameter returns an [invalid_index_uid](0061-error-format-and-definitions.md#invalid_index_uid) error. +- 🔴 If the requested `index_uid` does not exist, the API returns an [index_not_found](0061-error-format-and-definitions.md#index_not_found) error. + +#### 3.3.2. `PUT` - `indexes/:index_uid/settings/proximity-precision` + +Modify the `proximityPrecision` setting of a Meilisearch index. + +##### 3.3.2.1. Request Payload Definition + +- Type: String / `null` + +Setting `null` is equivalent to using the [3.3.3. `DELETE` - `indexes/:index_uid/settings/proximity-precision`](#333-delete---indexesindexuidsettingsproximity-precision) API endpoint. + +##### 3.3.2.2. Response Definition + +When the request is successful, Meilisearch returns the HTTP code `202 Accepted`. The response's content is the summarized representation of the received asynchronous task. + +See [Summarized `task` Object for `202 Accepted`](0060-tasks-api.md#summarized-task-object-for-202-accepted). + +##### 3.3.2.3. Errors + +- 🔴 Omitting Content-Type header returns a [missing_content_type](0061-error-format-and-definitions.md#missing_content_type) error. +- 🔴 Sending an empty Content-Type returns an [invalid_content_type](0061-error-format-and-definitions.md#invalid_content_type) error. +- 🔴 Sending a different Content-Type than `application/json` returns an [invalid_content_type](0061-error-format-and-definitions.md#invalid_content_type) error. +- 🔴 Sending an empty payload returns a [missing_payload](0061-error-format-and-definitions.md#missing_payload) error. +- 🔴 Sending an invalid JSON payload returns a [malformed_payload](0061-error-format-and-definitions.md#malformed_payload) error. +- 🔴 Sending an invalid index uid format for the `:index_uid` path parameter returns an [invalid_index_uid](0061-error-format-and-definitions.md#invalid_index_uid) error. +- 🔴 Sending a request payload value type different of a `String` or `null` returns an [invalid_settings_proximity_precision](0061-error-format-and-definitions.md#invalid_settings_proximity_precision) error. +- 🔴 Sending a request payload `String` that is not `byWord` or `byAttribute` returns an [invalid_settings_proximity_precision](0061-error-format-and-definitions.md#invalid_settings_proximity_precision) error. + +###### 3.3.2.3.1. Async Errors + +- 🔴 When Meilisearch is secured, if the API Key do not have the `indexes.create` action defined, the API returns an [index_not_found](0061-error-format-and-definitions.md#index_not_found) error in the related asynchronous `task` resource. See [3.3.2.2. Response Definition](#3222-response-definition). + +> Otherwise, Meilisearch will create the index in a lazy way. See [3.2.2.4. Lazy Index Creation](#3224-lazy-index-creation). + +##### 3.3.2.4. Lazy Index Creation + +If the requested `index_uid` does not exist, and the authorization layer allows it (See [3.3.2.3.1. Async Errors](#33231-async-errors)), Meilisearch will create the index when the related asynchronous task resource is executed. See [3.3.2.2. Response Definition](#3322-response-definition). + +#### 3.3.3. `DELETE` - `indexes/:index_uid/settings/proximity-precision` + +Reset the `proximityPrecision` setting of a Meilisearch index to the default value `null`. + +##### 3.3.3.1. Response Definition + +When the request is in a successful state, Meilisearch returns the HTTP code `202 Accepted`. The response's content is the summarized representation of the received asynchronous task. + +See [Summarized `task` Object for `202 Accepted`](0060-tasks-api.md#summarized-task-object-for-202-accepted). + +##### 3.3.3.3. Errors + +- 🔴 Sending an invalid index uid format for the `:index_uid` path parameter returns an [invalid_index_uid](0061-error-format-and-definitions.md#invalid_index_uid) error. + +###### 3.3.3.3.1. Asynchronous Index Not Found Error + +- 🔴 If the requested `index_uid` does not exist, the API returns an [index_not_found](0061-error-format-and-definitions.md#index_not_found) error in the related async `task` resource. See [3.3.3.1. Response Definition](#3331-response-definition). + +#### 3.3.4. General Errors + +These errors apply to all endpoints described here. + +##### 3.3.4.1 Auth Errors + +The auth layer can return the following errors if Meilisearch is secured (a master-key is defined). + +- 🔴 Accessing this route without the `Authorization` header returns a [missing_authorization_header](0061-error-format-and-definitions.md#missing_authorization_header) error. +- 🔴 Accessing this route with a key that does not have permissions (i.e. other than the master-key) returns an [invalid_api_key](0061-error-format-and-definitions.md#invalid_api_key) error. + +## 4. Technical Details + +### 4.1. Triggering Documents Re-Indexing + +Meilisearch favors search speed and makes a trade-off on indexing speed by computing internal data structures to get search results as fast as possible. + +Modifying this index setting cause documents to be re-indexed. \ No newline at end of file diff --git a/text/0123-settings-api.md b/text/0123-settings-api.md index 53e0198a..de143849 100644 --- a/text/0123-settings-api.md +++ b/text/0123-settings-api.md @@ -27,6 +27,7 @@ N/A | [typo-tolerance](0117-typo-tolerance-setting-api.md) | `typoTolerance` sub-resource API endpoints definition | | [pagination](157-pagination-setting-api.md) | `pagination` sub-resource API endpoints definition | | [faceting](157-faceting-setting-api.md) | `faceting` sub-resource API endpoints definition | +| [faceting](0123-proximity-precision.md) | `proximityPrecision` sub-resource API endpoints definition | Each setting is exposed as a sub-resource of the `indexes/:index_uid/settings` endpoints. e.g. The ranking rules setting of a Meilisearch index is exposed at `indexes/:index_uid/settings/ranking-rules`. @@ -58,6 +59,7 @@ Fetch the settings of a Meilisearch index. | `typoTolerance` | Object | true | | `pagination` | Object | true | | `faceting` | Object | true | +| `proximityPrecision` | String | true | The attributes ordering in the response payload is equivalent to the order described in the table above. @@ -87,6 +89,7 @@ Modify the settings of a Meilisearch index. | `typoTolerance` | Object / `null` | false | | `pagination` | Object / `null` | false | | `faceting` | Object / `null` | false | +| `proximityPrecision` | String / `null` | false | - The request payload accepts partial definitions, the value of missing fields will remain unchanged. - The request payload accepts the `null` value for any setting, which will reset the setting to its default value. @@ -169,6 +172,7 @@ Changing any of the following index settings will cause documents to be re-index - `separatorTokens` - `nonSeparatorTokens` - `dictionary` +- `proximityPrecision` ## 5. Future Possibilities diff --git a/text/0262-task-queue-webhook.md b/text/0262-task-queue-webhook.md new file mode 100644 index 00000000..a908049d --- /dev/null +++ b/text/0262-task-queue-webhook.md @@ -0,0 +1,56 @@ +# Task queue webhook + +## 1. Summary + +Describe the usage of a webhook URL that will be called whenever a task finishes so a third party can be notified. +It is used to compute some of the required metrics for the Monitoring Metrics v1 project. Heavily related to the +format of the [`tasks`](/text/0060-tasks-api.md#fully-qualified-task-object). + +## 2. Motivation + +It'll give users the possibility to wait on task completion _without_ having to poll meilisearch over and over again. +This heavily reduces the network congestion and will allow us to compute the different metrics/ charts needed for the +[Monitoring Metrics v1](https://www.notion.so/ Monitoring-Metrics-v1-4782d56795c043799dde33309e73a20f? pvs=21) project +that is related to the tasks queue (*Indexing latency, tasks operations*), we need to receive task information on the +cloud stack. + +## 3. Functional Specification + +### 3.1. Summary Key Points + +- The webhook URL is configured either via the env var `MEILI_TASK_WEBHOOK_URL` or via the CLI option flag `--task-webhook-url` + - By default, the webhook URL value is empty. + - If the given value does not match a valid URL format, a human-readable error is returned to the user, and the engine does not start. + - A valid user-provided URL is kept as is. + - You can optionally set an Authorization header by using the `--task-webhook-authorization-header` CLI option flag or via the `MEILI_TASK_WEBHOOK_AUTHORIZATION_HEADER` env var. +- When a valid webhook URL is configured and when a batch of tasks reaches a finished status (`status: succeeded/failed`), the engine sends a `POST` request to the user-configured webhook URL. The HTTP request contains the finished tasks in its payload. + - The sent payload is compressed with gzip and represented in the JSON Lines text format. +- A task sent over the webhook URL matches the [task object definition](https://github.com/meilisearch/specifications/blob/main/text/0060-tasks-api.md#1-task-object-definition). +- If an error is encountered while sending the payload to the URL, Meilisearch logs it but does not stop processing tasks. +- Payload example + - In that case, the user configured `https://myproject.com/mywebhook?common=people` +```json +//POST HTTP request to https://myproject.com/mywebhook?common=people + +{"uid":4,"indexUid":"movie","status":"failed","type":"indexDeletion","canceledBy":null,"details.deletedDocuments":0,"error.message":"Index `movie` not found.","error.code":"index_not_found","error.type":"invalid_request","error.link":"https://docs.meilisearch.com/errors#index_not_found","duration":"PT0.001192S","enqueuedAt":"2022-08-04T12:28:15.159167Z","startedAt":"2022-08-04T12:28:15.161996Z","finishedAt":"2022-08-04T12:28:15.163188Z"} +{"uid":5,"indexUid":"movie","status":"failed","type":"indexDeletion","canceledBy":null,"details.deletedDocuments":0,"error.message":"Index `movie` not found.","error.code":"index_not_found","error.type":"invalid_request","error.link":"https://docs.meilisearch.com/errors#index_not_found","duration":"PT0.001192S","enqueuedAt":"2022-08-04T12:28:15.159167Z","startedAt":"2022-08-04T12:28:15.161996Z","finishedAt":"2022-08-04T12:28:15.163188Z"} +{"uid":6,"indexUid":"movie","status":"failed","type":"indexDeletion","canceledBy":null,"details.deletedDocuments":0,"error.message":"Index `movie` not found.","error.code":"index_not_found","error.type":"invalid_request","error.link":"https://docs.meilisearch.com/errors#index_not_found","duration":"PT0.001192S","enqueuedAt":"2022-08-04T12:28:15.159167Z","startedAt":"2022-08-04T12:28:15.161996Z","finishedAt":"2022-08-04T12:28:15.163188Z"} +``` + +--- + +### 3.1. CLI Definition + +You can find the CLI information about the webhook [here](/text/0119-instance-options.md) + +## 4. Technical Aspects + +### 4.1. Sending a batch + +While a batch is being sent, the scheduler won't process a new batch until the webhook accepts the payload. + +## 5. Future Possibilities + +- Let users subscribe to a webhook to specific indexes or tasks instead of providing one global hook for everyone. +- Let users update the URL of their webhook at runtime. +- Stops blocking the processing of new tasks while we're sending a payload.