Speed up synthetic source #87882

nik9000 · 2022-06-21T11:57:58Z

This speeds up synthetic source, especially when there are many fields
in the index that are declared in the mapping but don't have values.
This is fairly common with ECS, and the tsdb rally track uses that. And
this improves fetch performance of that track:

|  50th percentile service time |    default |   6.24029 |  4.85568 | ms | -22.19% |
|  90th percentile service time |    default |   7.89923 |  6.52069 | ms | -17.45% |
|  99th percentile service time |    default |  12.0306  | 16.435   | ms | +36.61% |
| 100th percentile service time |    default |  14.2873  | 17.1175  | ms | +19.81% |
|  50th percentile service time | default_1k | 158.425   | 25.3236  | ms | -84.02% |
|  90th percentile service time | default_1k | 165.46    | 30.8655  | ms | -81.35% |
|  99th percentile service time | default_1k | 168.954   | 33.3342  | ms | -80.27% |
| 100th percentile service time | default_1k | 174.341   | 34.8344  | ms | -80.02% |

There's a slight increase in the 99th and 100th percentile service time
for fetching ten document which think is unlucky jitter. Hopefully. The
average performance of fetching ten docs improves anyway so I think
we're ok. Fetching a thousand documents improves 80% across the board
which is lovely.

This works by doing three things:

Teach the "leaf" layer of source loader to detect when the field is
empty in that segment and remove it from the synthesis process
entirely. This brings most of the speed up in tsdb.
Replace hasValue with advanceToDoc returning boolean and
cache the result.
Replace the ArrayList of leaf loaders with an array. Before fixing
the other two issues the ArrayList's iterator really showed up in
the profiling. Probably much less worth it now, but it's small.

All of this brings synthetic source much closer to the fetch performance
of standard _source:

|  50th percentile service time | default_1k |  11.4016  | 25.3236  | ms | +122.11% |
|  90th percentile service time | default_1k |  13.7212  | 30.8655  | ms | +124.95% |
|  99th percentile service time | default_1k |  15.8785  | 33.3342  | ms | +109.93% |
| 100th percentile service time | default_1k |  16.9715  | 34.8344  | ms | +105.25% |

One important thing, these perf numbers come from fetching hot blocks
on disk. They mostly compare CPU overhead and not disk overhead.

This speeds up synthetic source, especially when there are many fields in the index that are declared in the mapping but don't have values. This is fairly common with ECS, and the tsdb rally track uses that. And this improves fetch performance of that track: ``` | 50th percentile service time | default | 6.24029 | 4.85568 | ms | -22.19% | | 90th percentile service time | default | 7.89923 | 6.52069 | ms | -17.45% | | 99th percentile service time | default | 12.0306 | 16.435 | ms | +36.61% | | 100th percentile service time | default | 14.2873 | 17.1175 | ms | +19.81% | | 50th percentile service time | default_1k | 158.425 | 25.3236 | ms | -84.02% | | 90th percentile service time | default_1k | 165.46 | 30.8655 | ms | -81.35% | | 99th percentile service time | default_1k | 168.954 | 33.3342 | ms | -80.27% | | 100th percentile service time | default_1k | 174.341 | 34.8344 | ms | -80.02% | ``` There's a slight increase in the 99th and 100th percentile service time for fetching ten document which think is unlucky jitter. Hopefully. The average performance of fetching ten docs improves anyway so I think we're ok. Fetching a thousand documents improves 80% across the board which is lovely. This works by doing three things: 1. Teach the "leaf" layer of source loader to detect when the field is empty in that segment and remove it from the synthesis process entirely. This brings most of the speed up in tsdb. 2. Replace `hasValue` with a callback when writing the first value. `hasValue` was resulting in a 2^n-like number of calls that really showed up in the profiler. 3. Replace the `ArrayList` of leaf loaders with an array. Before fixing the other two issues the `ArrayList`'s iterator really showed up in the profiling. Probably much less worth it now, but it's small. All of this brings synthetic source much closer to the fetch performance of standard _source: ``` | 50th percentile service time | default_1k | 11.4016 | 25.3236 | ms | +122.11% | | 90th percentile service time | default_1k | 13.7212 | 30.8655 | ms | +124.95% | | 99th percentile service time | default_1k | 15.8785 | 33.3342 | ms | +109.93% | | 100th percentile service time | default_1k | 16.9715 | 34.8344 | ms | +105.25% | ``` One important thing, these perf numbers come from fetching *hot* blocks on disk. They mostly compare CPU overhead and not disk overhead.

elasticmachine · 2022-06-21T13:13:53Z

Pinging @elastic/es-analytics-geo (Team:Analytics)

nik9000 · 2022-06-21T13:14:00Z

labelbot I have set a label

server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java

romseygeek · 2022-06-21T13:27:40Z

server/src/main/java/org/elasticsearch/index/mapper/ObjectMapper.java

-                                    started = true;
-                                    startSyntheticField(b);
+                    public void load(XContentBuilder b, CheckedRunnable<IOException> before) throws IOException {
+                        class HasValue implements CheckedRunnable<IOException> {


This took me a few passes to understand, can we call it something like StartObjectEmitter?

Actually, dumb question - does this work if we change advanceToDoc() to return a boolean that says whether or not it is positioned? We have to call it on every subfield anyway, and then the object already knows whether or not it contains any values on the current doc before we get to load.

Let me have a look. I thought it wouldn't be man I would love to get rid of this thing.

Let me have a look. I thought it wouldn't be man I would love to get rid of this thing.

I... can't.... So close....

@romseygeek and I brainstormed and replaced the callbacks with a cached boolean. Simple enough.

nik9000 · 2022-06-21T17:42:54Z

@romseygeek this is ready for you again any time!

romseygeek

Thanks, this looks much nicer! A couple of nits but LGTM otherwise, no need for another review.

romseygeek · 2022-06-22T08:45:05Z

server/src/main/java/org/elasticsearch/index/mapper/SourceLoader.java

@@ -82,14 +82,24 @@ public boolean reordersFieldValues() {
        @Override
        public Leaf leaf(LeafReader reader) throws IOException {
            SyntheticFieldLoader.Leaf leaf = loader.leaf(reader);
+            if (leaf.empty()) {
+                return new Leaf() {


Given this has no state I wonder if it's worth having it as a final instance on Leaf?

romseygeek · 2022-06-22T08:46:03Z

server/src/main/java/org/elasticsearch/index/mapper/SourceLoader.java

@@ -104,20 +114,19 @@ public BytesReference source(FieldsVisitor fieldsVisitor, int docId) throws IOEx
     * Load a field for {@link Synthetic}.
     */
    interface SyntheticFieldLoader {
-        /**


Still worth having some javadoc on this I think?

Not sure what I did there.

romseygeek · 2022-06-22T08:46:22Z

server/src/main/java/org/elasticsearch/index/mapper/SourceLoader.java


            /**
-             * Load values for this document.
+             * Write values for this document.


elasticmachine · 2022-06-24T18:57:30Z

Pinging @elastic/es-search (Team:Search)

elasticsearchmachine added the v8.4.0 label Jun 21, 2022

nik9000 requested a review from romseygeek June 21, 2022 13:12

nik9000 added the >non-issue label Jun 21, 2022

nik9000 marked this pull request as ready for review June 21, 2022 13:12

nik9000 added the :StorageEngine/TSDB You know, for Metrics label Jun 21, 2022

elasticmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jun 21, 2022

nik9000 mentioned this pull request Jun 21, 2022

Synthetic Source #86603

Closed

50 tasks

romseygeek reviewed Jun 21, 2022

View reviewed changes

nik9000 added 2 commits June 21, 2022 10:03

Big comment

cbf22e5

Changes with Alan

ca099ec

nik9000 requested a review from romseygeek June 21, 2022 17:42

Rename method

9aebc5a

romseygeek approved these changes Jun 22, 2022

View reviewed changes

nik9000 added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Jun 22, 2022

nik9000 added 2 commits June 22, 2022 08:32

Fixup

e772fbf

Format

9762bd7

elasticsearchmachine merged commit 152d3e9 into elastic:master Jun 22, 2022

nik9000 deleted the synthetic_source_speed_obj branch June 22, 2022 13:52

javanna added the :Search Foundations/Mapping Index mappings, including merging and defining field types label Jun 24, 2022

elasticmachine added the Team:Search Meta label for search team label Jun 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speed up synthetic source #87882

Speed up synthetic source #87882

nik9000 commented Jun 21, 2022 •

edited

Loading

elasticmachine commented Jun 21, 2022

nik9000 commented Jun 21, 2022

romseygeek Jun 21, 2022

romseygeek Jun 21, 2022

nik9000 Jun 21, 2022

nik9000 Jun 21, 2022

nik9000 Jun 21, 2022

nik9000 commented Jun 21, 2022

romseygeek left a comment

romseygeek Jun 22, 2022

nik9000 Jun 22, 2022

romseygeek Jun 22, 2022

nik9000 Jun 22, 2022

romseygeek Jun 22, 2022

elasticmachine commented Jun 24, 2022

Speed up synthetic source #87882

Speed up synthetic source #87882

Conversation

nik9000 commented Jun 21, 2022 • edited Loading

elasticmachine commented Jun 21, 2022

nik9000 commented Jun 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 commented Jun 21, 2022

romseygeek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Jun 24, 2022

nik9000 commented Jun 21, 2022 •

edited

Loading