You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: learn/indexing/optimize_indexing_performance.mdx
+13-13Lines changed: 13 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,17 +1,17 @@
1
1
---
2
-
title: Optimize indexing performance by analyzing batch statistics
2
+
title: Optimize indexing performance with batch statistics
3
3
description: Learn how to analyze the `progressTrace` to identify and resolve indexing bottlenecks in Meilisearch.
4
4
---
5
5
6
6
# Optimize indexing performance by analyzing batch statistics
7
7
8
8
Indexing performance can vary significantly depending on your dataset, index settings, and hardware. The [batch object](/reference/api/batches) provides information about the progress of asynchronous indexing operations.
9
9
10
-
The `progressTrace` field within the batch object offers a detailed breakdown of where time is spent during the indexing process. By analyzing this data, you can identify bottlenecks and adjust configuration settings to improve indexing speed.
10
+
The `progressTrace` field within the batch object offers a detailed breakdown of where time is spent during the indexing process. Use this data to identify bottlenecks and improve indexing speed.
11
11
12
12
## Understanding the `progressTrace`
13
13
14
-
The `progressTrace` is a hierarchical trace showing each phase of indexing and how long it took.
14
+
`progressTrace` is a hierarchical trace showing each phase of indexing and how long it took.
15
15
Each entry follows the structure:
16
16
17
17
```json
@@ -24,15 +24,15 @@ This means:
24
24
- The subtask was **extracting word proximity**.
25
25
- It took **33.71 seconds**.
26
26
27
-
Your goal is to focus on the **longest-running steps** and understand which index settings or data characteristics influence them.
27
+
Focus on the **longest-running steps** and investigate which index settings or data characteristics influence them.
28
28
29
29
## Key phases and how to optimize them
30
30
31
31
### Document processing
32
32
33
33
| Trace key | Description | Optimization |
34
34
|------------|--------------|--------------|
35
-
|`computing document changes`, `extracting documents`| Meilisearch compares incoming documents to existing ones. | No direct optimization possible. The duration scales with the number and size of incoming documents.|
35
+
|`computing document changes`, `extracting documents`| Meilisearch compares incoming documents to existing ones. | No direct optimization possible. Process duration scales with the number and size of incoming documents.|
36
36
37
37
### Filterable attributes
38
38
@@ -44,40 +44,40 @@ Your goal is to focus on the **longest-running steps** and understand which inde
44
44
45
45
| Trace key | Description | Optimization |
46
46
|------------|--------------|--------------|
47
-
|`extracting words`, `merging word caches`| Tokenizes text and builds the inverted index. |- Ensure the [**searchable attributes**](/reference/api/settings#searchable-attributes) list includes only the fields you want to be checked for query word matches. |
47
+
|`extracting words`, `merging word caches`| Tokenizes text and builds the inverted index. | Ensure the [searchable attributes](/reference/api/settings#searchable-attributes) list only includes the fields you want to be checked for query word matches. |
48
48
49
49
### Proximity precision
50
50
51
51
| Trace key | Description | Optimization |
52
52
|------------|--------------|--------------|
53
-
|`extracting word proximity`, `merging word proximity`| Builds the data structures for phrase and attribute ranking. | Lower the precision of this operation by setting [proximity precision](/reference/api/settings#proximity-precision) to `byAttribute`instead of the default `byWord`|
53
+
|`extracting word proximity`, `merging word proximity`| Builds data structures for phrase and attribute ranking. | Lower the precision of this operation by setting [proximity precision](/reference/api/settings#proximity-precision) to `byAttribute`|
54
54
55
55
### Disk I/O and hardware bottlenecks
56
56
57
57
| Trace key | Description | Optimization |
58
58
|------------|--------------|--------------|
59
-
|`waiting for database writes`| Time spent writing data to disk. | No direct optimization possible. Either the disk is slow, either the quantity of data to write is big. Avoid HDDs (Hard Disk Drives).|
59
+
|`waiting for database writes`| Time spent writing data to disk. | No direct optimization possible. Either the disk is too slow or you are writing too much data in a single operation. Avoid HDDs (Hard Disk Drives) |
60
60
|`waiting for extractors`| Time spent waiting for CPU-bound extraction. | No direct optimization possible. Indicates a CPU bottleneck. Use more cores or scale horizontally with [sharding](/learn/advanced/sharding). |
61
61
62
62
### Facets and filterable attributes
63
63
64
64
| Trace key | Description | Optimization |
65
65
|------------|--------------|--------------|
66
-
|`post processing facets > strings bulk` / `numbers bulk`| Processes equality or comparison filters. | - Disable unused [**filter features**](/reference/api/settings#features), such as comparison operators on string values. <br/>- Keep [**sortable attributes**](reference/api/settings#sortable-attributes) to the minimum required. |
66
+
|`post processing facets > strings bulk` / `numbers bulk`| Processes equality or comparison filters. | - Disable unused [**filter features**](/reference/api/settings#features), such as comparison operators on string values. <br /> - Reduce the number of [**sortable attributes**](reference/api/settings#sortable-attributes). |
67
67
|`post processing facets > facet search`| Builds structures for the [facet search API](/reference/api/facet_search). | If you don’t use the facet search API, [disable it](/reference/api/settings#update-facet-search-settings).|
68
68
69
69
### Embeddings
70
70
71
71
| Trace key | Description | Optimization |
72
72
|------------|--------------|--------------|
73
-
|`writing embeddings to database`| Time spent saving vector embeddings. |- Use smaller embedding vectors when possible. <br/>- You can avoid recomputing embeddings on document update by [disabling embedding regeneration](/reference/api/documents#vectors). <br/>- Consider enabling [binary quantization](/reference/api/settings#binaryquantized) for your embedders. |
73
+
|`writing embeddings to database`| Time spent saving vector embeddings. | Use embedding vectors with fewer dimensions. <br/>- [Disabling embedding regeneration on document update](/reference/api/documents#vectors). <br/>- Consider enabling [binary quantization](/reference/api/settings#binaryquantized). |
74
74
75
75
### Word prefixes and post-processing
76
76
77
77
| Trace key | Description | Optimization |
78
78
|------------|--------------|--------------|
79
-
|`post processing words > word prefix *`| Builds prefix data for autocomplete. Allows to match documents that begin with a specific query term, instead of only exact matches.| Disable [**prefix search**](/reference/api/settings#prefix-search) (`prefixSearch: disabled`) if not required. <br/> Note that this can severely impact search result relevancy. |
80
-
|`post processing words > word fst`| Builds the word FST (finite state transducer). | No direct action possible, as it depends on the number of different words in the database. Fewer searchable words can improve speed. |
79
+
|`post processing words > word prefix *`| Builds prefix data for autocomplete. Allows matching documents that begin with a specific query term, instead of only exact matches.| Disable [**prefix search**](/reference/api/settings#prefix-search) (`prefixSearch: disabled`). *This can severely impact search result relevancy.*|
80
+
|`post processing words > word fst`| Builds the word FST (finite state transducer). | No direct action possible, as FST size reflect the number of different words in the database. Using documents with fewer searchable words may improve operation speed. |
The [facet search feature](/learn/filtering_and_sorting/search_with_facet_filters#searching-facet-values) is consuming significant time. If your application doesn’t use it, disable it:
90
+
[Facet searching](/learn/filtering_and_sorting/search_with_facet_filters#searching-facet-values) is raking significant indexing time. If your application doesn’t use facets, disable the feature:
0 commit comments