Failing ES Promotion: FTR Configs #25 / visualize app visual builder tsdb check "before all" hook for "should render from a tsdb dataView regular fields with no issues"

[mistic]

Chrome UI Functional Tests
test/functional/apps/visualize/group6/_tsvb_tsdb_basic.ts

visualize app visual builder tsdb check "before all" hook for "should render from a tsdb dataView regular fields with no issues"

This failure is preventing the promotion of the current Elasticsearch snapshot.

For more information on the Elasticsearch snapshot process including how to reproduce using the unverified ES build please read the failed promotion annotation. Other important information can be found at:

• Failed promotion job
• Test Failure

AggregateError: 
    Error: Bulk doc failure [operation=index]:
      doc: {"@timestamp":"2023-06-15T14:36:26.463Z","agent":"Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24","bytes":206,"bytes_counter":78427335,"bytes_gauge":206,"clientip":"24.98.25.38","event":{"dataset":"sample_web_logs"},"extension":"","geo":{"coordinates":{"lat":41.992934150621295,"lon":-86.12801251932979},"dest":"TD","src":"US","srcdest":"US:TD"},"host":"www.elastic.co","index":"kibana_sample_data_logs","ip":"24.98.25.38","machine":{"os":"win 7","ram":15032385536},"message":"24.98.25.38 - - [2018-09-20T14:36:26.463Z] /"GET / HTTP/1.1/" 200 206 /"-/" /"Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24/"","referer":"http://nytimes.com/success/valery-bykovsky","request":"/","response":"200","tags":["info","success"],"url":"https://www.elastic.co/products","utc_time":"2023-06-15T14:36:26.463Z"}
      error: {"type":"document_parsing_exception","reason":"[1:905] failed to parse: _id must be unset or set to [PPl4MT9wr3nRjFh0AAABiL98e98] but was [PPl4MYGl8aNGtSVXAAABiL98e98] because [kibana_sample_data_logstsdb] is in time_series mode","caused_by":{"type":"illegal_argument_exception","reason":"_id must be unset or set to [PPl4MT9wr3nRjFh0AAABiL98e98] but was [PPl4MYGl8aNGtSVXAAABiL98e98] because [kibana_sample_data_logstsdb] is in time_series mode"}}
        at Array.map (<anonymous>)
        at indexDocs (index_doc_records_stream.ts:64:13)
        at processTicksAndRejections (node:internal/process/task_queues:95:5)
        at Writable.write [as _write] (index_doc_records_stream.ts:78:9)
    at indexDocs (index_doc_records_stream.ts:64:13)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Writable.write [as _write] (index_doc_records_stream.ts:78:9)

[elasticmachine]

Pinging @elastic/kibana-visualizations (Team:Visualizations)

[mistic]

Skipped.

main: 7c45bfd

[stratoula]

@dej611 can you check this as it is tsdb related? 🙏

[salvatore-campagna]

Trying to guess here what is happening.

You have a dataset saved in some storage that includes documents with an _id filed and a specific value.
You load the dataset and index it (probably a sample dataset).
Note that setting the _id field in time_series mode is not allowed because of routing reasons and constraints about where the document is indexed.
When you load the dataset and index it you probably index the whole document including the _id.
You should index the document without specifying the _id.

If you ask "why is this happening now but not before?" I guess the answer is "because before it was matching the expected value". Now ids are different because we changed the way the _tsid is generated and the _tsid is used as input to calculate the document _id.

[salvatore-campagna]

See a similar issue elastic/elasticsearch#101212