Skip to content
This repository has been archived by the owner on Mar 21, 2024. It is now read-only.

Search API - Formatting Search Results #120

Merged
merged 33 commits into from
May 13, 2022
Merged

Conversation

gmourier
Copy link
Member

@gmourier gmourier commented Mar 1, 2022

Naming proposals are open until 2022-03-31
The definitive specification is due at 2022-04-01


Summary

This specification describes the parameters that can be used to format search results with Meilisearch in a native way.

It also deletes an old specification dedicated to the _formatted field and reorganizes the sections of the specification concerning search endpoints.

🤖 API Diff
📄 Rendered


Key Points

Additions

Customize Highlight Tags

  • highlightPreTag: (Type: String, Required: false, Default: "<em>")
  • highlightPostTag: (Type: String, Required: false, Default: "</em>")

Customize Default Crop Marker

  • cropMarker: (Type: String, Required: false, Default: )

Behaviors Adjustments

Crop around words instead of characters

  • cropLength is now a number of words.
  • The cropMarker is affixed "smartly" around the cropped part of an attribute.

Cropped part selection

The cropping algorithm tries to match the window with the highest density of query terms within the cropLength limit. Then it will pick the window that contains the more ordered query terms.

Out-of-scope

Unchanged

  • matches and _matchesInfo remain unchanged.

Future Possibilities

  • All parameters related to the formatting of search results could be under a formatter object.
  • formatter could be exposed as an index setting to be overridable at search time.

Informations To Reviewers

API Design Questions

Naming

Consulting @meilisearch/integration-team and @meilisearch/docs-team about the naming for the new parameters.

  • highlightPreTag
  • highlightPostTag
  • cropMarker

Misc

  • Update OpenAPI specification file
  • Describes future possibilities
  • Merge filter-and-facet-behavior spec into dedicated field description of the search spec

@gmourier gmourier added In Progress Feature specification is in elaboration. Important changes can still occurs in the specification. OpenAPI Update OpenAPI specification. Q2:2022 v0.27 labels Mar 1, 2022
@gmourier gmourier self-assigned this Mar 7, 2022
@gmourier gmourier added the P2 label Mar 7, 2022
@gmourier gmourier force-pushed the formatting-search-results branch 4 times, most recently from 6e2525f to 840ea98 Compare March 9, 2022 11:56
@github-actions
Copy link

github-actions bot commented Mar 9, 2022

🤖 API change detected:

Modified (2)

  • GET /indexes/{indexUid}/search
    • Query parameters added: highlightPreTag, highlightPostTag, cropMarker
  • POST /indexes/{indexUid}/search
    • Body modified
      • Body attributes added: highlightPreTag, highlightPostTag, cropMarker

View documentation diff

Powered by Bump

@gmourier gmourier force-pushed the formatting-search-results branch from 4ae1c41 to abcbdd0 Compare March 9, 2022 14:11
@gmourier gmourier added Ready For Review Feature specification must be reviewed. and removed In Progress Feature specification is in elaboration. Important changes can still occurs in the specification. labels Mar 9, 2022
@gmourier gmourier marked this pull request as ready for review March 9, 2022 14:15
@gmourier gmourier added the Patch This PR patch a feature specification label Mar 9, 2022
@gmourier gmourier changed the title Formatting Search Results Search API - Formatting Search Results Mar 9, 2022
irevoire
irevoire previously approved these changes Mar 9, 2022
text/0118-search-api.md Outdated Show resolved Hide resolved
text/0118-search-api.md Outdated Show resolved Hide resolved
text/0118-search-api.md Show resolved Hide resolved
text/0118-search-api.md Outdated Show resolved Hide resolved
text/0118-search-api.md Outdated Show resolved Hide resolved
text/0118-search-api.md Outdated Show resolved Hide resolved
text/0118-search-api.md Outdated Show resolved Hide resolved
text/0118-search-api.md Outdated Show resolved Hide resolved
text/0118-search-api.md Show resolved Hide resolved
text/0118-search-api.md Outdated Show resolved Hide resolved
@gmourier gmourier added P1 and removed P2 labels Mar 10, 2022
@gmourier gmourier force-pushed the formatting-search-results branch 3 times, most recently from 6044e59 to e452a16 Compare March 15, 2022 18:05
gmourier and others added 18 commits May 13, 2022 12:22
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
…/ highlightPreTag on attributesToCrop & attributesToHighlight
Co-authored-by: gui machiavelli <hey@guimachiavelli.com>
… it keep the phrase context when extending around
Co-authored-by: Many <legendre.maxime.isn@gmail.com>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
Co-authored-by: Tommy <68053732+dichotommy@users.noreply.github.com>
Co-authored-by: gui machiavelli <hey@guimachiavelli.com>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
* Improve formatted spec

* Update text/0118-search-api.md

Co-authored-by: Tamo <irevoire@protonmail.ch>

* Update text/0118-search-api.md

Co-authored-by: Tamo <irevoire@protonmail.ch>

* Update text/0118-search-api.md

Co-authored-by: Tamo <irevoire@protonmail.ch>

* Update text/0118-search-api.md

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Update text/0118-search-api.md

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Improve according to reviews

Co-authored-by: Tamo <irevoire@protonmail.ch>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
@gmourier gmourier force-pushed the formatting-search-results branch from 7603eee to 38ba16d Compare May 13, 2022 10:23
@gmourier gmourier merged commit 8f7d585 into develop May 13, 2022
@gmourier gmourier deleted the formatting-search-results branch May 13, 2022 10:36
gmourier added a commit that referenced this pull request May 13, 2022
* Draft a revamp of the formatting search results spec

* Removes formatting-search-results specification

* wip

* Fix links

* Add precision

* Update OpenAPI

* Fix sentences

* Add consistency

* Add details and fix sentences

* Fix default value in OpenAPI spec

* Add Future Possibilities for _matchesInfo

* Remove bullet point title

* Add wip examples

* Add _formatted behavior regarding attributesToRetrieve, attributesToCrop and attributesToHighlight

* Apply suggestions from code review

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Fix typo

* Mention no effect case of  cropMarker / cropLength / highlightPostTag / highlightPreTag  on attributesToCrop  & attributesToHighlight

* Re-explain _formatted in details

* Mention Synonyms

* Add precision after team feedback

* Apply suggestions from code review

Co-authored-by: gui machiavelli <hey@guimachiavelli.com>

* Add precisions

* Add boolean analytics for fields dedicated to customize formatting of search results behaviors

* Precise behavior for _formatted

* Add clearer explanations for the cropping algorithm and the fact that it keep the phrase context when extending around

* Apply suggestions from code review

Co-authored-by: Many <legendre.maxime.isn@gmail.com>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
Co-authored-by: Tommy <68053732+dichotommy@users.noreply.github.com>

* Precise highlighting around every matched query term

* Apply suggestions from code review

Co-authored-by: gui machiavelli <hey@guimachiavelli.com>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Precise cropLength behavior when 0 is specified

* Remove filter-and-facet-behavior spec to merge it into search-api, and correct some types description

* Improve formatted spec (#146)

* Improve formatted spec

* Update text/0118-search-api.md

Co-authored-by: Tamo <irevoire@protonmail.ch>

* Update text/0118-search-api.md

Co-authored-by: Tamo <irevoire@protonmail.ch>

* Update text/0118-search-api.md

Co-authored-by: Tamo <irevoire@protonmail.ch>

* Update text/0118-search-api.md

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Update text/0118-search-api.md

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Improve according to reviews

Co-authored-by: Tamo <irevoire@protonmail.ch>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Add missing precision for highlightPostTag

* fix broken links

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
Co-authored-by: gui machiavelli <hey@guimachiavelli.com>
Co-authored-by: Many <legendre.maxime.isn@gmail.com>
Co-authored-by: Tommy <68053732+dichotommy@users.noreply.github.com>
Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>
Co-authored-by: Tamo <irevoire@protonmail.ch>
@gmourier gmourier added Implemented Feature specification has been implemented. and removed P1 Ready For Review Feature specification must be reviewed. labels May 13, 2022
gmourier added a commit that referenced this pull request May 16, 2022
* Instance Options (#119)

* Add instance options spec

* Update text/0119-instance-options.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Update text/0119-instance-options.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Update text/0119-instance-options.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Update text/0119-instance-options.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Update text/0119-instance-options.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Update text/0119-instance-options.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Update text/0119-instance-options.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Update text/0119-instance-options.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Update text/0119-instance-options.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Numeric title + add dump options

* Add link to dump spec

* Update text/0119-instance-options.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Update text/0119-instance-options.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Add title numerotations

* Add telemetry for new options

* Apply changes after the review

* Update text/0119-instance-options.md

* Update text/0119-instance-options.md

Co-authored-by: Many <legendre.maxime.isn@gmail.com>

* Update text/0119-instance-options.md

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Update text/0119-instance-options.md

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Change naming after the review

* Update text/0034-telemetry-policies.md

* Update text/0034-telemetry-policies.md

* Update text/0034-telemetry-policies.md

* Update text/0034-telemetry-policies.md

* Apply suggestions from code review

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Update text/0119-instance-options.md

Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>

* Update text/0119-instance-options.md

Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>
Co-authored-by: Many <legendre.maxime.isn@gmail.com>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Settings API - v0.26.0 state (#123)

* Add a Settings API specification file

* Add distinct-attribute-setting-api.md

* Add displayed-attributes and searchable-attributes specification file

* Add stop-words setting api

* Add synonyms setting api

* Rename spec files and fix some typos

* Add ranking rules setting api

* Add filterable-attributes and sortable-attributes settings API

* Add a Triggering Documents Re-Indexing technical section

* Rephrase 202 Accepted Response for POST Methods

* Apply Triggering Documents Re-Indexing Technical part on related sub settings

* Update synonyms setting api file

* Precise and rephrase some sentences

* Remove bullet point title

* Add limitation about manually specifying searchableAttributes

* Update spec template according to our current usage (#127)

* Update spec template according to our current usage

* Update text/0000-specification-template.md

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Remove bullet point dedicated to title

* Rename Technical Aspects to Technical Details

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>

* Documents API (#133)

* Catch-up documents api specification

* Add precisions and examples

* Apply suggestions from code review

Co-authored-by: Many <legendre.maxime.isn@gmail.com>

Co-authored-by: Many <legendre.maxime.isn@gmail.com>

* Stats API (#134)

* Add Stats API specification

* Rename spec file with pr number

* Clean typo, add precisions and fix inner linking

* Apply suggestions from code review

Co-authored-by: maryamsulemani97 <90181761+maryamsulemani97@users.noreply.github.com>

* Change order of API endpoints

Co-authored-by: maryamsulemani97 <90181761+maryamsulemani97@users.noreply.github.com>

* Errors - Add variant for `invalid_filter`/`invalid_sort` when related settings are empty (#125)

* Add variant for invalid_sort and invalid_filter errors when settings are empty

* Update text/0061-error-format-and-definitions.md

Co-authored-by: gui machiavelli <hey@guimachiavelli.com>

Co-authored-by: gui machiavelli <hey@guimachiavelli.com>

* Add missing "v" in the user-agent data (#136)

Following the issue meilisearch/integration-guides#150 there is a "v" before the version of the package.

* Indexes API (#132)

* Catch-up Indexes API

* Add details by field for an Index API Resource

* fix typo

* Add future possibilities

* Apply suggestions from code review

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Add precisions from review

* lowercase type

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Remove never-implemented and deprecated 0042-paginated-search.md (#147)

* Settings API - Typo Tolerance (#117)

* Draft file for Typo Tolerance specification

* rename spec file

* Rename specification

* draft

* Add open-api.yaml prototype

* Add typoTolerance component on POST search

* Add typoTolerance object into search api parameters

* fix typos

* remove the past tense

* Add typoTolerance GET query parameters

* Fix description

* Add explanations for disableOnWords property

* Update spec href

* Describe indexes/:index_uid/settings/typo-tolerance endpoint

* Explain why it is useful to expose typo tolerance settings at search time

* Remove typoTolerance properties at search time

* Add a future possiblity to change POST verb to PATCH to edit a settings partially

* Sync spec with telemetry

* Mark TODO on technical details and rework structure of specification

* Specify properties required to false to allow partial updates

* Add error definitions for typoTolerance API resource properties

* Rename spec file

* Restore deleted metric

* Update text/0117-typo-tolerance-settings-api.md

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Update text/0117-typo-tolerance-settings-api.md

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Bruno Casali <brunoocasali@gmail.com>

* Replace W to w in telemtry

* Apply consistency on new errors message

* fix typo

* Add Async Errors and Lazy Index Creation

* Rename telemetry nodes

* Mention case insensivity of disableOnWords parameter

* Describes synchronous (type-checking)  and asynchronous (business logic) errors

* Add examples for minWordSizeFor1Typo and minWordSizeFor2Typos

* Elaborate on examples

* Remove title bullet point

* Apply naming suggestions

* Apply naming changes on telemetry and miss on OpenAPI

* Branch typo sub-ressource to settings-api specification

* Fix typo (lul)

* Add Technical Details

* Apply suggestions from code review

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Replace minWordSizeForXTypos fields by a minWordSizeForTypos object

* Update text/0034-telemetry-policies.md

Co-authored-by: ad hoc <postma.marin@protonmail.com>

* Apply suggestions from code review

Co-authored-by: Tommy <68053732+dichotommy@users.noreply.github.com>

* Remove dedicated error for type checking

* Update text/0034-telemetry-policies.md

Co-authored-by: Bruno Casali <brunoocasali@gmail.com>

* Bring naming changes

* Fix metrics name for typo_tolerance

* Update text/0034-telemetry-policies.md

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
Co-authored-by: Bruno Casali <brunoocasali@gmail.com>
Co-authored-by: ad hoc <postma.marin@protonmail.com>
Co-authored-by: Tommy <68053732+dichotommy@users.noreply.github.com>

* Data Types - Add Nested Fields Support (#121)

* Add a draft spec with nested fields flattening examples

* Consitent phrasing

* Fix markdown syntax

* Add null section

* Add Object example section

* Apply suggestions from code review

Co-authored-by: Clément Renault <renault.cle@gmail.com>

* Rename spec file to PR id

* Add special searchableAttributes default case with flattening alg

* Rephrase document structure representation

* Add a dotnotation section

* Add dot-notation section

* Update text/0121-data-types.md

Co-authored-by: Quentin de Quelen <quentin@meilisearch.com>

* Add Future Possibilities

* Rework nested notation sections

* Fix TODO and mention the . notation and all properties notation

* Add a edge case section

* Fix missing block code marker

* Update text/0123-ranking-rules-setting-api.md

Co-authored-by: Clément Renault <renault.cle@gmail.com>

* Apply suggestions from code review

Co-authored-by: Tommy <68053732+dichotommy@users.noreply.github.com>
Co-authored-by: Clément Renault <renault.cle@gmail.com>

* Fix section number

* Remove unclear sentence

* Remove dedicated object section

Co-authored-by: Clément Renault <renault.cle@gmail.com>
Co-authored-by: Quentin de Quelen <quentin@meilisearch.com>
Co-authored-by: Tommy <68053732+dichotommy@users.noreply.github.com>

* Search API - Formatting Search Results (#120)

* Draft a revamp of the formatting search results spec

* Removes formatting-search-results specification

* wip

* Fix links

* Add precision

* Update OpenAPI

* Fix sentences

* Add consistency

* Add details and fix sentences

* Fix default value in OpenAPI spec

* Add Future Possibilities for _matchesInfo

* Remove bullet point title

* Add wip examples

* Add _formatted behavior regarding attributesToRetrieve, attributesToCrop and attributesToHighlight

* Apply suggestions from code review

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Fix typo

* Mention no effect case of  cropMarker / cropLength / highlightPostTag / highlightPreTag  on attributesToCrop  & attributesToHighlight

* Re-explain _formatted in details

* Mention Synonyms

* Add precision after team feedback

* Apply suggestions from code review

Co-authored-by: gui machiavelli <hey@guimachiavelli.com>

* Add precisions

* Add boolean analytics for fields dedicated to customize formatting of search results behaviors

* Precise behavior for _formatted

* Add clearer explanations for the cropping algorithm and the fact that it keep the phrase context when extending around

* Apply suggestions from code review

Co-authored-by: Many <legendre.maxime.isn@gmail.com>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
Co-authored-by: Tommy <68053732+dichotommy@users.noreply.github.com>

* Precise highlighting around every matched query term

* Apply suggestions from code review

Co-authored-by: gui machiavelli <hey@guimachiavelli.com>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Precise cropLength behavior when 0 is specified

* Remove filter-and-facet-behavior spec to merge it into search-api, and correct some types description

* Improve formatted spec (#146)

* Improve formatted spec

* Update text/0118-search-api.md

Co-authored-by: Tamo <irevoire@protonmail.ch>

* Update text/0118-search-api.md

Co-authored-by: Tamo <irevoire@protonmail.ch>

* Update text/0118-search-api.md

Co-authored-by: Tamo <irevoire@protonmail.ch>

* Update text/0118-search-api.md

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Update text/0118-search-api.md

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Improve according to reviews

Co-authored-by: Tamo <irevoire@protonmail.ch>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>

* Add missing precision for highlightPostTag

* fix broken links

Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
Co-authored-by: gui machiavelli <hey@guimachiavelli.com>
Co-authored-by: Many <legendre.maxime.isn@gmail.com>
Co-authored-by: Tommy <68053732+dichotommy@users.noreply.github.com>
Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>
Co-authored-by: Tamo <irevoire@protonmail.ch>

* Bump API to v0.27.0 version

Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>
Co-authored-by: Many <legendre.maxime.isn@gmail.com>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
Co-authored-by: maryamsulemani97 <90181761+maryamsulemani97@users.noreply.github.com>
Co-authored-by: gui machiavelli <hey@guimachiavelli.com>
Co-authored-by: Bruno Casali <brunoocasali@gmail.com>
Co-authored-by: ad hoc <postma.marin@protonmail.com>
Co-authored-by: Tommy <68053732+dichotommy@users.noreply.github.com>
Co-authored-by: Clément Renault <renault.cle@gmail.com>
Co-authored-by: Quentin de Quelen <quentin@meilisearch.com>
Co-authored-by: Tamo <irevoire@protonmail.ch>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Implemented Feature specification has been implemented. OpenAPI Update OpenAPI specification. Patch This PR patch a feature specification Q2:2022 Telemetry Update the telemetry collect. v0.27
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants