Releases: meilisearch/meilisearch
v1.11.1 🐿️
What's Changed
Full Changelog: v1.11.0...v1.11.1
v1.11.0 🐿️
Meilisearch v1.11 introduces AI-powered search performance improvements thanks to binary quantization and various usage changes, all of which are steps towards a future stabilization of the feature. We have also improved federated search usage following user feedback.
🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment happens between 4 to 48 hours after a new version becomes available.
Some SDKs might not include all new features. Consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️).
New features and updates 🔥
Experimental - AI-powered search improvements
This release is Meilisearch's first step towards stabilizing AI-powered search and introduces a few breaking changes to its API. Consult the PRD for full usage details.
Done by @dureuill in #4906, #4920, #4892, and #4938.
⚠️ Breaking changes
- When performing AI-powered searches,
hybrid.embedder
is now a mandatory parameter inGET
andPOST
/indexes/{:indexUid}/search
- As a consequence, it is now mandatory to pass
hybrid
even for pure semantic searches embedder
is now a mandatory parameter inGET
andPOST
/indexes/{:indexUid}/similar
- Meilisearch now ignores
semanticRatio
and performs a pure semantic search for queries that includevector
but notq
Addition & improvements
- The default model for OpenAI is now
text-embedding-3-small
instead oftext-embedding-ada-002
- This release introduces a new embedder option:
documentTemplateMaxBytes
. Meilisearch will truncate a document's template text when it goes over the specified limit - Fields in
documentTemplate
include a newfield.is_searchable
property. The default document template now filters out both empty fields and fields not in the searchable attributes list:
v1.11:
{% for field in fields %}
{% if field.is_searchable and not field.value == nil %}
{{ field.name }}: {{ field.value }}\n
{% endif %}
{% endfor %}
v1.10:
{% for field in fields %}
{{ field.name }}: {{ field.value }}\n
{% endfor %}
Embedders using the v1.10 document template will continue working as before. The new default document template will only work with newly created embedders.
Vector database indexing performance improvements
v1.11 introduces a new embedder option, binaryQuantized
:
curl \
-X PATCH 'http://localhost:7700/indexes/movies/settings' \
-H 'Content-Type: application/json' \
--data-binary '{
"embedders": {
"image2text": {
"binaryQuantized": true
}
}
}'
Enable binary quantization to convert embeddings of floating point numbers into embeddings of boolean values. This will negatively impact the relevancy of AI-powered searches but significantly improve performance in large collections with more than 100 dimensions.
In our benchmarks, this reduced the size of the database by a factor of 10 and divided the indexing time by a factor of 6 with little impact on search times.
Warning
Enabling this feature will update all of your vectors to contain only 1
s or -1
s, significantly impacting relevancy.
You cannot revert this option once you enable it. Before setting binaryQuantized
to true
, Meilisearch recommends testing it in a smaller or duplicate index in a development environment.
Federated search improvements
Facet distribution and stats for federated searches
This release adds two new federated search options, facetsByIndex
and mergeFacets
. These allow you to request a federated search for facet distributions and stats data.
Facet information by index
To obtain facet distribution and stats for each separate index, use facetsByIndex
when querying the POST
/multi-search
endpoint:
POST /multi-search
{
"federation": {
"limit": 20,
"offset": 0,
"facetsByIndex": {
"movies": ["title", "id"],
"comics": ["title"],
}
},
"queries": [
{
"q": "Batman",
"indexUid": "movies"
},
{
"q": "Batman",
"indexUid": "comics"
}
]
}
The multi-search response will include a new field, facetsByIndex
with facet data separated per index:
{
"hits": […],
…
"facetsByIndex": {
"movies": {
"distribution": {
"title": {
"Batman returns": 1
},
"id": {
"42": 1
}
},
"stats": {
"id": {
"min": 42,
"max": 42
}
}
},
…
}
}
Merged facet information
To obtain facet distribution and stats for all indexes merged into a single, use both facetsByIndex
and mergeFacets
when querying the POST
/multi-search
endpoint:
POST /multi-search
{
"federation": {
"limit": 20,
"offset": 0,
"facetsByIndex": {
"movies": ["title", "id"],
"comics": ["title"],
},
"mergeFacets": {
"maxValuesPerFacet": 10,
}
}
"queries": [
{
"q": "Batman",
"indexUid": "movies"
},
{
"q": "Batman",
"indexUid": "comics"
}
]
}
The response includes two new fields, facetDistribution
and facetStarts
:
{
"hits": […],
…
"facetDistribution": {
"title": {
"Batman returns": 1
"Batman: the killing joke":
},
"id": {
"42": 1
}
},
"facetStats": {
"id": {
"min": 42,
"max": 42
}
}
}
Experimental — New STARTS WITH
filter operator
Enable the experimental feature to use the STARTS WITH
filter operator:
curl \
-X PATCH 'http://localhost:7700/experimental-features/' \
-H 'Content-Type: application/json' \
--data-binary '{
"containsFilter": true
}'
Use the STARTS WITH
operator when filtering:
curl \
-X POST http://localhost:7700/indexes/movies/search \
-H 'Content-Type: application/json' \
--data-binary '{
"filter": "hero STARTS WITH spider"
}'
🗣️ This is an experimental feature, and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.
Done by @Kerollmops in #4939.
Other improvements
- Language support and localizedAttributes settings by @ManyTheFish in #4937
- Add ISO-639-1 variants
- Convert ISO-639-1 into ISO-639-3
- Add a German language tokenizer by @luflow in meilisearch/charabia#303 and in #4945
- Improve Turkish language support by @tkhshtsh0917 in meilisearch/charabia#305 and in #4957
- Upgrade "batch failed" log to error level in #4955 by @dureuill.
- Update the search UI: remove the forced capitalized fields, by @curquiza in #4993
Fixes 🐞
⚠️ When using federated search,query.facets
was silently ignored at the query level, but should not have been. It now returns the appropriate error. Usefederation.facetsByIndex
instead if you want facets to be applied during federated search.- Prometheus
/metrics
return the route pattern instead of the real route when returning the HTTP requests total by @irevoire in #4839 - Truncate values at the end of a list of facet values when the number of facet values is larger than
maxValuesPerFacet
. For example, settingmaxValuesPerFacet
to2
could result in["blue", "red", "yellow"]
, being truncated to["blue", "yellow"]
instead of ["blue", "red"]`. By @dureuill in #4929 - Improve the task cancellation when vectors are used, by @irevoire in #4971
- Swedish support: the characters
å
,ä
,ö
are no longer normalized toa
ando
. By @ManyTheFish in #4945 - Update rhai to fix an internal error when updating documents with a function (experimental) by @irevoire in #4960
- Fix the bad experimental search queue size by @irevoire in #4992
- Do not send empty edit document by function by @irevoire in #5001
- Display vectors when no custom vectors were ever provided by @dureuill in #5008
Misc
- Dependencies updates
- Security dependency upgrade: bump quinn-proto from 0.11.3 to 0.11.8 by @dependabot in #4911
- CIs and tests
- Documentation
- Fix broken links in README by @iornstein in #4943
- Misc
❤️ Thanks again to our external contributors:
v1.11.0-rc.3 🐿️
Warning
Since this is a release candidate (RC), we do NOT recommend using it in a production environment. Is something not working as expected? We welcome bug reports and feedback about new features.
Maintenance
v1.11.0-rc.2 🐿️
Warning
Since this is a release candidate (RC), we do NOT recommend using it in a production environment. Is something not working as expected? We welcome bug reports and feedback about new features.
Bugs Fixes
- fix the bad experimental search queue size by @irevoire in #4992
- Do not send empty edit document by function by @irevoire in #5001
- Display vectors when no custom vectors were ever provided by @dureuill in #5008
Experimental Features
See the release note of the v1.10.3 for more information about the two experimental cli flags that are being introduced in this RC.
Misc
Full Changelog: v1.11.0-rc.1...v1.11.0-rc.2
v1.10.3 🦩
Search improvements
This PR lets you configure two behaviors of the engine through experimental cli flags:
- The number of searches Meilisearch can process concurrently per core with the
--experimental-nb-searches-per-core
cli flag - After how many seconds Meilisearch can consider a search as irrelevant and drop it straight away without processing it with the
--experimental-drop-search-after
cli flag
Full Changelog: v1.10.2...v1.10.3
v1.11.0-rc.1 🐿️
Warning
Since this is a release candidate (RC), we do NOT recommend using it in a production environment. Is something not working as expected? We welcome bug reports and feedback about new features.
What's Changed
- Update rhai by @irevoire in #4960
This should fix a bug where Meilisearch was reporting an internal error while modifying documents. If you encounter the same issue, please try the RC and let us know if it fixes the issue! - Update charabia feature flags and add support for the Turkish language by @ManyTheFish in #4957
- Update arroy by @irevoire in #4971
- Make the indexing of vectors slightly quicker
- Improve the cancelling of task containing vectors
Misc
- Upgrade "batch failed" log to error level by @dureuill in #4955
- Fix bench by adding embedder by @dureuill in #4954
- Move the multi arroy index logic to the arroy wrapper by @irevoire in #4953
- Improve the error message by @dureuill in #4972
Full Changelog: v1.11.0-rc.0...v1.11.0-rc.1
v1.10.2 🦩
Fixes 🦋
Activate the Swedish tokenization Pipeline
The Swedish tokenization pipeline were deactivated in the previous versions, now it is activated when specifying the index Language in the settings:
PATCH /indexes/:index-name/settings
{
"localizedAttributes": [ { "locales": ["swe"], "attributePatterns": ["*"] } ]
}
related PR: #4949
v1.11.0-rc.0 🐿️
Warning
Since this is a release candidate (RC), we do NOT recommend using it in a production environment. Is something not working as expected? We welcome bug reports and feedback about new features.
Meilisearch v1.11 introduces AI-powered search performance improvements thanks to binary quantization and various usage changes, all of which are steps towards a future stabilization of the feature. We have also improved federated search usage following user feedback.
New features and updates 🔥
Experimental - AI-powered search improvements
This release is Meilisearch's first step towards the stabilization of AI-powered search and introduces a few breaking changes to its API. Consult the PRD for full usage details.
Done by @dureuill in #4906, #4920, #4892, and #4938.
⚠️ Breaking changes
- When performing AI-powered searches,
hybrid.embedder
is now a mandatory parameter inGET
andPOST
/indexes/{:indexUid}/search
- As a consequence, it is now mandatory to pass
hybrid
even for pure semantic searches embedder
is now a mandatory parameter inGET
andPOST
/indexes/{:indexUid}/similar
- Meilisearch now ignores
semanticRatio
and performs a pure semantic search for queries that includevector
but notq
Addition & improvements
- The default model for OpenAI is now
text-embedding-3-small
instead oftext-embedding-ada-002
- This release introduces a new embedder option:
documentTemplateMaxBytes
. Meilisearch will truncate a document's template text when it goes over the specified limit - Fields in
documentTemplate
include a newfield.is_searchable
property. It is also possible to filter out document fields:
v1.11:
{% for field in fields %}
{% if field.is_searchable and not field.value == nil %}
{{ field.name }}: {{ field.value }}\n
{% endif %}
{% endfor %}
v1.10:
{% for field in fields %}
{{ field.name }}: {{ field.value }}\n
{% endfor %}
Changes to document templates are backwards compatible. Embedders using the v1.10 document template will continue working as before The new default document template will only work newly created embedders.
Vector database indexing performance improvements
v1.11 introduces a new embedder option, binaryQuantized
:
curl \
-X PATCH 'http://localhost:7700/indexes/movies/settings' \
-H 'Content-Type: application/json' \
--data-binary '{
"embedders": {
"image2text": {
"binaryQuantized": true
}
}
}'
Enable binary quantization to convert embeddings of floating point numbers into embeddings of boolean values. This will negatively impact the relevancy of AI-powered searches, but significantly improve performance in large collections with more than 100 dimensions.
In our benchmarks, this reduced the size of the database by a factor of 10, and divided the indexing time by a factor of 6 with little impact on search times.
Warning
Enabling this feature will update all of your vectors to contain only 1
s or -1
s, impacting the relevancy significantly.
You cannot revert this option once you enable it. Before setting binaryQuantized
to true
, Meilisearch recommends testing it in a smaller or duplicate index in a development environment.
Federated search improvements
Facet distribution and stats for federated searches
This release adds two new federated search options, facetsByIndex
and mergeFacets
. These allow you to request facet distributions and stats data for a federated search.
Facet information by index
To obtain facet distribution and stats for each separate index, use facetsByIndex
when querying the POST
/multi-search
endpoint:
POST /multi-search
{
"federation": {
"limit": 20,
"offset": 0,
"facetsByIndex": {
"movies": ["title", "id"],
"comics": ["title"],
}
},
"queries": [
{
"q": "Batman",
"indexUid": "movies"
},
{
"q": "Batman",
"indexUid": "comics"
}
]
}
The multi-search response will include a new field, facetsByIndex
with facet data separated per index:
{
"hits": […],
…
"facetsByIndex": {
"movies": {
"distribution": {
"title": {
"Batman returns": 1
},
"id": {
"42": 1
}
},
"stats": {
"id": {
"min": 42,
"max": 42
}
}
},
…
}
}
Merged facet information
To obtain facet distribution and stats for all index merged into a single ,use both facetsByIndex
and mergeFacets
when querying the POST
/multi-search
endpoint:
POST /multi-search
{
"federation": {
"limit": 20,
"offset": 0,
"facetsByIndex": {
"movies": ["title", "id"],
"comics": ["title"],
},
"mergeFacets": {
"maxValuesPerFacet": 10,
}
}
"queries": [
{
"q": "Batman",
"indexUid": "movies"
},
{
"q": "Batman",
"indexUid": "comics"
}
]
}
The response includes two new fields, facetDistribution
and facetStarts
:
{
"hits": […],
…
"facetDistribution": {
"title": {
"Batman returns": 1
"Batman: the killing joke":
},
"id": {
"42": 1
}
},
"facetStats": {
"id": {
"min": 42,
"max": 42
}
}
}
Experimental — New STARTS_WITH
filter operator
Enable the experimental feature to use the STARTS_WITH
filter operator:
curl \
-X PATCH 'http://localhost:7700/experimental-features/' \
-H 'Content-Type: application/json' \
--data-binary '{
"containsFilter": true
}'
Use the STARTS_WITH
operator when filtering:
curl \
-X POST http://localhost:7700/indexes/movies/search \
-H 'Content-Type: application/json' \
--data-binary '{
"filter": "hero STARTS_WITH spider"
}'
🗣️ This is an experimental feature, and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.
Done by @Kerollmops in #4939.
Other improvements
- Language support and localizedAttributes settings by @ManyTheFish in #4937
- Add ISO-639-1 variants
- Convert ISO-639-1 into ISO-639-3
- Add a German language tokenizer by @luflow in meilisearch/charabia#303 and in #4945
Fixes 🐞
⚠️ When using federated search,query.facets
was silently ignored at the query level, but should not have been. It now returns the appropriate error. Usefederation.facetsByIndex
instead if you want facets to be applied during federated search.- Prometheus
/metrics
return the route pattern instead of the real route when returning the HTTP requests total by @irevoire in #4839 - Truncate values at the end of a list of facet values when the number of facet values is larger than
maxValuesPerFacet
. For example, settingmaxValuesPerFacet
to2
could result in["blue", "red", "yellow"]
, being truncated to["blue", "yellow"]
instead of ["blue", "red"]`. By @dureuill in #4929 - Swedish support: the characters
å
,ä
,ö
are no longer normalized toa
ando
. By @ManyTheFish in #4945
Misc
- Dependencies updates
- Security dependency upgrade: bump quinn-proto from 0.11.3 to 0.11.8 by @dependabot in #4911
- CIs and tests
- Documentation
- Fix broken links in README by @iornstein in #4943
- Misc
❤️ Thanks again to our external contributors:
v1.10.1 🦩
Fixes 🦋
Better search handling under heavy loads
All of the next PR should make meilisearch behave better under heavy loads:
- Only spawn one search queue in actix-web by @irevoire in #4893
- Make sure the index scheduler never stops running by @irevoire in #4896
- Explicitly drop the search permits by @irevoire in #4898
- Stop trying to process searches after one minute by @irevoire in #4899
Speed improvement 🐎
We made the autobatching of the document deletion with the document deletion by filter possible which should uncklog the task queue of the people using these two operations heavily.
Meilisearch still cannot autobatch the document deletion by filter and the document addition, though.
- Autobatch document deletion by filter by @irevoire in #4901
- Do not fail the whole batch when a single document deletion by filter fails by @irevoire in #4905
Full Changelog: v1.10.0...v1.10.1
v1.9.1 🦎
Fixes 🪲
- Return an empty list of embeddings for embedders that have no document for an embedder. by @dureuill in #4889
This fixes an issue where dumps created for indexes with:
- A user-provided embedder
- At least one documents that opt-out of vectors for that user-provided embedder
would fail to import correctly.
Upgrade path to v1.10.0 🚀
If you are a Cloud user affected by the above issue, please contact customer support so we perform the upgrade for you.
If you are an OSS user affected by the above, perform the following operations:
- Upgrade from v1.9.0 to v1.9.1 without using a dump
- Upgrade to v1.10.0 using a dump created from v1.9.1