ddl: Fix vector index for high dimensional vectors #58717

breezewish · 2025-01-06T09:20:51Z

What problem does this PR solve?

Issue Number: Close #58836

Problem Summary:

What changed and how does it work?

Fix Txn: KeyTooLong error when using vector index. It is caused by TiDB writes not only vector data but also vector keys, but TiDB shouldn't, as vector indexes are TiFlash local indexes and there is no need to write vector keys.

In this PR, the vector_long cluster integration test is just using the same test case as TestVectorLong unit test, simply run over the whole cluster to ensure everything works.

This is a cherry-pick of upstream PR: https://github.com/tidbcloud/tidb-cse/pull/1464

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: Wish <breezewish@outlook.com>

codecov · 2025-01-06T09:39:16Z

Codecov Report

Attention: Patch coverage is 90.47619% with 2 lines in your changes missing coverage. Please review.

Project coverage is 75.2373%. Comparing base (510d003) to head (c46a2b9).
Report is 7 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #58717        +/-   ##
================================================
+ Coverage   73.0776%   75.2373%   +2.1596%     
================================================
  Files          1676       1724        +48     
  Lines        463741     478773     +15032     
================================================
+ Hits         338891     360216     +21325     
+ Misses       103993      95978      -8015     
- Partials      20857      22579      +1722

Flag	Coverage Δ
integration	`48.8700% <10.5263%> (?)`
unit	`72.4921% <78.9473%> (+0.2339%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`61.6613% <ø> (+15.9313%)`	⬆️

Signed-off-by: Wish <breezewish@outlook.com>

breezewish · 2025-01-08T12:46:37Z

/test fast_test_tiprow

ti-chi-bot · 2025-01-08T12:46:41Z

@breezewish: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build

/test check-dev

/test check-dev2

/test mysql-test

/test pull-br-integration-test

/test pull-integration-ddl-test

/test pull-integration-e2e-test

/test pull-lightning-integration-test

/test pull-mysql-client-test

/test pull-unit-test-ddlv1

/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test

/test pull-common-test

/test pull-e2e-test

/test pull-integration-common-test

/test pull-integration-copr-test

/test pull-integration-jdbc-test

/test pull-integration-mysql-test

/test pull-integration-nodejs-test

/test pull-integration-python-orm-test

/test pull-scan-deps

/test pull-sqllogic-test

/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build

pingcap/tidb/ghpr_check

pingcap/tidb/ghpr_check2

pingcap/tidb/ghpr_mysql_test

pingcap/tidb/ghpr_unit_test

pingcap/tidb/pull_br_integration_test

pingcap/tidb/pull_integration_ddl_test

pingcap/tidb/pull_integration_e2e_test

pingcap/tidb/pull_lightning_integration_test

pingcap/tidb/pull_mysql_client_test

pingcap/tidb/pull_unit_test_ddlv1

In response to this:

/test fast_test_tiprow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2025-01-08T12:46:55Z

@ti-chi-bot[bot]: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

@breezewish: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:
/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-integration-e2e-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test
The following commands are available to trigger optional jobs:
/test pingcap/tidb/canary_ghpr_unit_test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-scan-deps
/test pull-sqllogic-test
/test pull-tiflash-test
Use /test all to run the following jobs that were automatically triggered:
pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_br_integration_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_integration_e2e_test
pingcap/tidb/pull_lightning_integration_test
pingcap/tidb/pull_mysql_client_test
pingcap/tidb/pull_unit_test_ddlv1
In response to this:

/test fast_test_tiprow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

…long-vector * commit '510d0037b18f258f505abc6cf13a8128563e9359': *: upgrade pd client to make sure tso client initiate successfully (#58752) ttl, test: scale TTL workers during the fault tests (#58750) planner: improve warning messages for unsupported HASH_JOIN hints (#58646) planner: prealloc the slices in the SplitCorColAccessCondFromFilters (#58785) ddl: supports non-unique global index (#58678) util/stmtsummary: add the network traffic related fields (#58101) var: enable `pd_enable_follower_handle_region` as default (#58385) statistics: refactor stats meta handling to use DeltaUpdate for multi-table support (#58657) parser: move 'model' to 'ast' pkg (#58704) statistics: add recover to protect background task (#58739) disttask: cancel subtask context if scheduled away (#58615) *: don't handle live updates of column size (#58596) *: fix a bug for default_authentication_plugin (2) (#58723) dupdetect: gRPC cancel should trigger retry (#58542) *: fix a bug for default_authentication_plugin (#57391) distsql: Fix backoff execution info inaccurate issue (#58707)

Signed-off-by: Wish <breezewish@outlook.com>

breezewish · 2025-01-08T14:24:05Z

/retest

ti-chi-bot · 2025-01-09T06:00:23Z

[LGTM Timeline notifier]

Timeline:

2025-01-06 09:33:14.956885395 +0000 UTC m=+173338.245717100: ☑️ agreed by tangenta.
2025-01-09 06:00:22.870866023 +0000 UTC m=+419766.159697727: ☑️ agreed by lance6716.

ti-chi-bot · 2025-01-09T06:23:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lance6716, tangenta, XuHuaiyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/ddl/OWNERS~~ [lance6716,tangenta]
~~pkg/expression/OWNERS~~ [XuHuaiyu]
~~pkg/meta/OWNERS~~ [lance6716,tangenta]
~~pkg/table/OWNERS~~ [tangenta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

breezewish · 2025-01-09T06:32:56Z

/merge

ti-chi-bot · 2025-01-09T06:32:58Z

@breezewish: We have migrated to builtin LGTM and approve plugins for reviewing.

👉 Please use /approve when you want approve this pull request.

The changes announcement: LGTM plugin changes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

breezewish · 2025-01-09T11:06:40Z

/cherry-pick release-8.5

ti-chi-bot · 2025-01-09T11:07:22Z

@breezewish: new pull request created to branch release-8.5: #58835.

In response to this:

/cherry-pick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

close #58836

ddl: Fix vector index for long vectors

4033911

Signed-off-by: Wish <breezewish@outlook.com>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 6, 2025

tangenta approved these changes Jan 6, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jan 6, 2025

Fix bazel

8a776af

Signed-off-by: Wish <breezewish@outlook.com>

breezewish added 2 commits January 8, 2025 21:56

Fix tests

c46a2b9

Signed-off-by: Wish <breezewish@outlook.com>

lance6716 approved these changes Jan 9, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 9, 2025

XuHuaiyu approved these changes Jan 9, 2025

View reviewed changes

ti-chi-bot bot added the approved label Jan 9, 2025

ti-chi-bot bot merged commit 448e302 into master Jan 9, 2025
27 checks passed

ti-chi-bot bot deleted the wenxuan/long-vector branch January 9, 2025 07:31

ti-chi-bot mentioned this pull request Jan 9, 2025

ddl: Fix vector index for high dimensional vectors (#58717) #58835

Merged

13 tasks

breezewish mentioned this pull request Jan 10, 2025

planner: KeyTooLong when inserting VECTOR with 3000 dimension #58836

Closed

ti-chi-bot bot pushed a commit that referenced this pull request Jan 10, 2025

ddl: Fix vector index for high dimensional vectors (#58717) (#58835)

0005334

close #58836

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: Fix vector index for high dimensional vectors #58717

ddl: Fix vector index for high dimensional vectors #58717

breezewish commented Jan 6, 2025 •

edited

Loading

codecov bot commented Jan 6, 2025 •

edited

Loading

breezewish commented Jan 8, 2025

ti-chi-bot bot commented Jan 8, 2025

tiprow bot commented Jan 8, 2025

breezewish commented Jan 8, 2025

ti-chi-bot bot commented Jan 9, 2025

ti-chi-bot bot commented Jan 9, 2025

breezewish commented Jan 9, 2025

ti-chi-bot bot commented Jan 9, 2025

breezewish commented Jan 9, 2025

ti-chi-bot commented Jan 9, 2025

ddl: Fix vector index for high dimensional vectors #58717

ddl: Fix vector index for high dimensional vectors #58717

Conversation

breezewish commented Jan 6, 2025 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

codecov bot commented Jan 6, 2025 • edited Loading

Codecov Report

breezewish commented Jan 8, 2025

ti-chi-bot bot commented Jan 8, 2025

tiprow bot commented Jan 8, 2025

breezewish commented Jan 8, 2025

ti-chi-bot bot commented Jan 9, 2025

[LGTM Timeline notifier]

ti-chi-bot bot commented Jan 9, 2025

breezewish commented Jan 9, 2025

ti-chi-bot bot commented Jan 9, 2025

breezewish commented Jan 9, 2025

ti-chi-bot commented Jan 9, 2025

breezewish commented Jan 6, 2025 •

edited

Loading

codecov bot commented Jan 6, 2025 •

edited

Loading