Skip to content

Commit

Permalink
add lowercase tokenizer docs (#8497) (#8910)
Browse files Browse the repository at this point in the history
* add lowercase tokenizer docs



* Doc review



* Apply suggestions from code review




---------







(cherry picked from commit 68a8af5)

Signed-off-by: Anton Rubin <anton.rubin@eliatra.com>
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Fanit Kolchina <kolchfa@amazon.com>
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
  • Loading branch information
5 people authored Dec 9, 2024
1 parent be2f9a5 commit d8f6190
Showing 1 changed file with 93 additions and 0 deletions.
93 changes: 93 additions & 0 deletions _analyzers/tokenizers/lowercase.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
layout: default
title: Lowercase
parent: Tokenizers
nav_order: 70
---

# Lowercase tokenizer

The `lowercase` tokenizer breaks text into terms at white space and then lowercases all the terms. Functionally, this is identical to configuring a `letter` tokenizer with a `lowercase` token filter. However, using a `lowercase` tokenizer is more efficient because the tokenizer actions are performed in a single step.

## Example usage

The following example request creates a new index named `my-lowercase-index` and configures an analyzer with a `lowercase` tokenizer:

```json
PUT /my-lowercase-index
{
"settings": {
"analysis": {
"tokenizer": {
"my_lowercase_tokenizer": {
"type": "lowercase"
}
},
"analyzer": {
"my_lowercase_analyzer": {
"type": "custom",
"tokenizer": "my_lowercase_tokenizer"
}
}
}
}
}
```
{% include copy-curl.html %}

## Generated tokens

Use the following request to examine the tokens generated using the analyzer:

```json
POST /my-lowercase-index/_analyze
{
"analyzer": "my_lowercase_analyzer",
"text": "This is a Test. OpenSearch 123!"
}
```
{% include copy-curl.html %}

The response contains the generated tokens:

```json
{
"tokens": [
{
"token": "this",
"start_offset": 0,
"end_offset": 4,
"type": "word",
"position": 0
},
{
"token": "is",
"start_offset": 5,
"end_offset": 7,
"type": "word",
"position": 1
},
{
"token": "a",
"start_offset": 8,
"end_offset": 9,
"type": "word",
"position": 2
},
{
"token": "test",
"start_offset": 10,
"end_offset": 14,
"type": "word",
"position": 3
},
{
"token": "opensearch",
"start_offset": 16,
"end_offset": 26,
"type": "word",
"position": 4
}
]
}
```

0 comments on commit d8f6190

Please sign in to comment.