Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

syncer: support filter row by SQL expression (#1782) #1832

Merged

Conversation

ti-chi-bot
Copy link
Member

This is an automated cherry-pick of #1782

What problem does this PR solve?

support filter row changes by SQL expression

What is changed and how it works?

add an item in configuration files, example.

expression-filter:
  expr-1:
    schema: "expr_filter"
    table: "t1"
    insert-value-expr: "c_null IS NOT NULL"

note that schema and table don't support pattern matching.

parse the expression into SQL AST, using TiDB's AST rewriter to generate executor AST, evaluate each row changes to decide passing it.

Check List

Tests

  • white-box: check can handle data types that go-mysql returns to DM
  • every MySQL data type can be handled
  • can filter by INSERT, old and new value of UDPATE, DELETE
  • works during DDL, dropping the filtering column makes the filter disabled, adding the column back enables it again
  • works on generated column
  • the timestamp value in expression should explicitly specify timezone c_timestamp = '2021-01-01 12:34:56.5678+08:00'
  • works on SQL functions ts > NOW()
  • works on multiple column a*a + b*b = c*c
  • benchmark
    sysbench using DM master branch c2a7514 and TiDB v5.1.0: CPS 28k ~ 30k, left mountain in picture 1
    Using DM this PR a591b16, not enable expression filter, CPS 28k ~ 29k, middle mountain in picture 1
    Using DM this PR a591b16, filter 1% INSERT of 1 of 4 tables, CPS 27k ~ 29k, right mountain in picture 1
    Using DM this PR a591b16, filter 99% INSERT of 1 of 4 tables, CPS 27k ~ 30k, picture 2

image

image

Code changes

  • Has exported function/method change
  • Has exported variable/fields change
  • Has persistent data change

Side effects

  • Possible performance regression
  • Increased code complexity

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation
  • Need to be included in the release note

@ti-chi-bot ti-chi-bot added needs-update-docs Should update docs after this PR is merged. Remove this label once the docs are updated needs-update-release-note This PR should be added into release notes. Remove this label once the release notes are updated size/XXL type/cherry-pick-for-release-2.0 labels Jul 5, 2021
@ti-chi-bot ti-chi-bot requested a review from lichunzhu July 5, 2021 07:50
@ti-chi-bot ti-chi-bot added the status/LGT1 One reviewer already commented LGTM label Jul 5, 2021
Copy link
Collaborator

@GMHDBJD GMHDBJD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot
Copy link
Member Author

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • GMHDBJD
  • lance6716

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels Jul 6, 2021
@lance6716
Copy link
Collaborator

/merge

@ti-chi-bot
Copy link
Member Author

This pull request has been accepted and is ready to merge.

Commit hash: c7bedb6

@ti-chi-bot ti-chi-bot merged commit f9ccf36 into pingcap:release-2.0 Jul 6, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
needs-update-docs Should update docs after this PR is merged. Remove this label once the docs are updated needs-update-release-note This PR should be added into release notes. Remove this label once the release notes are updated size/XXL status/can-merge status/LGT2 Two reviewers already commented LGTM, ready for merge type/cherry-pick-for-release-2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants