Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix decode error when "NULL" value in the column with "primary key" flag #5879

Merged
merged 17 commits into from
Sep 16, 2022

Conversation

JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Sep 14, 2022

What problem does this PR solve?

Issue Number: close #5859

Problem Summary:
TiFlash assumes all values in the column with "primary key" flag are non-null value.

However, users could create such "NULL" value in the "primary key" column by following steps

-- create table with `source` is nullable
create table test.t_case (
`case_no` varchar(32) not null,
`source` varchar(20) default null,
`p` varchar(12) DEFAULT NULL,
primary key (`case_no`)
);

-- insert some data and left `source` to be empty
insert into test.t_case(case_no)
values ("1"), ("2"), ("3"), ("4");

-- drop the primary key, fill the `source` to be non-empty
alter table t_case drop primary key;
update test.t_case set `source` = '' where `source` is NULL;
-- add new primary key with case_no and source
alter table t_case add primary key (case_no, source);

-- send the snapshot data to tiflash
alter table t_case set tiflash replica 1;

What is changed and how it works?

  • mainly change the logic in addDefaultValueToColumnIfPossible
    • For non-clustered index table(is_common_handle=false && pk_is_handle=false), fill default value even if the column contains primary key flag when the column is not appear in the encoded value(those rows are not supposed to be return to users)
    • For clustered index/pk_is_handle table, ignore the primary key column as they are supposed to be decoded from the encoded key
  • Refine the exception message in insertRangeFrom
  • Add more region info when exception thrown in preHandleSnapshot/ingestSST

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Fix the panic issue after creating the primary index with a column containing `NULL` value

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 14, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • hongyunyan
  • lidezhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added needs-cherry-pick-release-4.0 PR which needs to be cherry-picked to release-4.0 needs-cherry-pick-release-5.0 PR which needs to be cherry-picked to release-5.0 do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-cherry-pick-release-5.1 PR which needs to be cherry-picked to release-5.1 release-note Denotes a PR that will be considered when it comes time to generate release notes. needs-cherry-pick-release-5.2 PR which needs to be cherry-picked to release-5.2 needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. needs-cherry-pick-release-6.0 Type: Need cherry pick to release-6.0 needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. needs-cherry-pick-release-6.2 size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 14, 2022
@JaySon-Huang JaySon-Huang changed the title [WIP] Decode error when "NULL" value in the column with "primary key" flag [WIP] FIx decode error when "NULL" value in the column with "primary key" flag Sep 14, 2022
@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

@JaySon-Huang JaySon-Huang changed the title [WIP] FIx decode error when "NULL" value in the column with "primary key" flag Fix decode error when "NULL" value in the column with "primary key" flag Sep 14, 2022
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 14, 2022
@JaySon-Huang
Copy link
Contributor Author

/cc @lidezhu @hongyunyan

@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Sep 15, 2022

The helper functions in RegionBlockReaderTestFixture is buggy when testing with primary key columns. I will split it into another PR to refine the test suite first. #5899

Signed-off-by: JaySon-Huang <tshent@qq.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5934.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Sep 16, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5935.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Sep 16, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5936.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Sep 16, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5937.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Sep 16, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Sep 16, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5938.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Sep 16, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5939.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5940.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Sep 16, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@JaySon-Huang JaySon-Huang deleted the fix_pk_with_null_value branch September 16, 2022 14:00
JaySon-Huang added a commit to ti-chi-bot/tiflash that referenced this pull request Sep 20, 2022
JaySon-Huang added a commit to ti-chi-bot/tiflash that referenced this pull request Sep 26, 2022
JaySon-Huang added a commit to ti-chi-bot/tiflash that referenced this pull request Sep 26, 2022
JaySon-Huang added a commit to ti-chi-bot/tiflash that referenced this pull request Sep 26, 2022
JaySon-Huang pushed a commit to JaySon-Huang/tiflash that referenced this pull request Sep 27, 2022
JaySon-Huang added a commit that referenced this pull request Oct 31, 2022
* DDL: Use Column Name Instead of Offset to Find the common handle cluster index (#5166) (#5191)

close #5154

* tests: Fix RegionBlockReaderTest helper functions (#5899) (#5924)

ref #5859

* Fix decode error when "NULL" value in the column with "primary key" flag (#5879) (#5932)

close #5859

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cherry-pick-release-4.0 PR which needs to be cherry-picked to release-4.0 needs-cherry-pick-release-5.0 PR which needs to be cherry-picked to release-5.0 needs-cherry-pick-release-5.1 PR which needs to be cherry-picked to release-5.1 needs-cherry-pick-release-5.2 PR which needs to be cherry-picked to release-5.2 needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. needs-cherry-pick-release-6.0 Type: Need cherry pick to release-6.0 needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. needs-cherry-pick-release-6.2 release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Decode error when "NULL" value in the column with "primary key" flag
7 participants