Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

executor: Add insert/replace ignore/on duplicate key support for local temporary table #26636

Merged
merged 17 commits into from
Aug 4, 2021

Conversation

lcwangchao
Copy link
Collaborator

@lcwangchao lcwangchao commented Jul 27, 2021

What problem does this PR solve?

Issue Number: close #26083

What is changed and how it works?

Add support for some clause of local temporary table:

  • insert ignore into ...
  • insert into ... on duplicated key ...
  • replace into ...

Check List

Tests

  • Unit test

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jul 27, 2021

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • djshow832
  • tiancaiamao

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 27, 2021
@lcwangchao lcwangchao marked this pull request as draft July 27, 2021 06:32
@ti-chi-bot ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 27, 2021
@lcwangchao lcwangchao changed the title executor: Add insert/replace ignore/on duplicate key support executor: Add insert/replace ignore/on duplicate key support for local temporary table Jul 27, 2021
@github-actions github-actions bot added the sig/execution SIG execution label Jul 27, 2021
@lcwangchao
Copy link
Collaborator Author

/run-check_dev_2

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jul 27, 2021
Copy link
Contributor

@tiancaiamao tiancaiamao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is much more difficult than I first thought.

if err = prefetchDataCache(ctx, txn, toBeCheckedRows); err != nil {
return err
// Temporary table need not to do prefetch because its all data are stored in the memory.
if e.Table.Meta().TempTableType == model.TempTableNone {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of change the code here, could we change the prefetchDataCache to make it a dummy operation when it's on the temporary table?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both is ok to me. The reason why put the if statement outside is that the function prefetchDataCache cannot access the table meta.

@@ -403,14 +403,9 @@ func (e *PointGetExecutor) get(ctx context.Context, key kv.Key) ([]byte, error)
// fallthrough to snapshot get.
}

// Global temporary table is always empty, so no need to send the request.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the global temporary table code is modified?

Copy link
Collaborator Author

@lcwangchao lcwangchao Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's my fault... Line 407 should be modified to e.tblInfo.TempTableType != model.TempTableNone. It's a draft and waiting for other prs to be merged first... so some tests are not added yet.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you find a bug here, it's better to file another PR for the bug.
Otherwise please reset this change ...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, there is no bug in previous code. Just because after we make TemporaryTableSnapshotReader can read both local and global temporary data, it can be more elegant after this modification. But it's still ok to revert it.

executor/point_get.go Show resolved Hide resolved
@lcwangchao
Copy link
Collaborator Author

/run-check_dev_2

@lcwangchao lcwangchao added the sig/sql-infra SIG: SQL Infra label Jul 30, 2021
@lcwangchao lcwangchao marked this pull request as ready for review July 30, 2021 10:30
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 30, 2021
@lcwangchao
Copy link
Collaborator Author

/run-check_dev_2

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 3, 2021
return err
// Temporary table need not to do prefetch because its all data are stored in the memory.
if e.Table.Meta().TempTableType == model.TempTableNone {
if _, err = prefetchUniqueIndices(ctx, txn, toBeCheckedRows); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The question is what is the return of perfetchUniqueIndices. If it returns an empty data, it seems not fit the semantics of the method. If it returns the data in temporary table, it seems unnecessary and no one depends the return to do anything...

executor/insert.go Show resolved Hide resolved
@@ -403,14 +403,9 @@ func (e *PointGetExecutor) get(ctx context.Context, key kv.Key) ([]byte, error)
// fallthrough to snapshot get.
}

// Global temporary table is always empty, so no need to send the request.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you find a bug here, it's better to file another PR for the bug.
Otherwise please reset this change ...

@tiancaiamao
Copy link
Contributor

I'm wondering if we can make the changes simpler.
You know, all the special hacking are decreasing readability...

For example:

// Temporary table need not to do prefetch because its all data are stored in the memory.
if e.Table.Meta().TempTableType != model.TempTableNone {
		return nil
}

values, err := prefetchUniqueIndices(ctx, txn, rows)

Why do we need to consider a special case for temporary table?

func (e *InsertValues) txnValueGetter(txn kv.Transaction) kv.Getter {
	tblInfo := e.Table.Meta()
	if tblInfo.TempTableType == model.TempTableNone {
		return txn
	}

	return e.ctx.GetSessionVars().TemporaryTableTxnReader(txn, tblInfo)
}
// Change txn => kvGetter   in many place...

What if we make perfetch do nothing for the temporary table (it doesn't matter as perfetch is only a optimization),
and use a txn that can handle the temporary table and its snapshot ...

The philosophy is that special logic make the code harder to maintain.

The normal transaction update process code is critical for general cases, it's not worthy to sacrifice the readability on behalf of non-general code like the temporary table.

(I'm not sure we can really achieve that goal ... just some personal thoughts.)

@lcwangchao
Copy link
Collaborator Author

I'm wondering if we can make the changes simpler.
You know, all the special hacking are decreasing readability...

For example:

// Temporary table need not to do prefetch because its all data are stored in the memory.
if e.Table.Meta().TempTableType != model.TempTableNone {
		return nil
}

values, err := prefetchUniqueIndices(ctx, txn, rows)

Why do we need to consider a special case for temporary table?

func (e *InsertValues) txnValueGetter(txn kv.Transaction) kv.Getter {
	tblInfo := e.Table.Meta()
	if tblInfo.TempTableType == model.TempTableNone {
		return txn
	}

	return e.ctx.GetSessionVars().TemporaryTableTxnReader(txn, tblInfo)
}
// Change txn => kvGetter   in many place...

What if we make perfetch do nothing for the temporary table (it doesn't matter as perfetch is only a optimization),
and use a txn that can handle the temporary table and its snapshot ...

The philosophy is that special logic make the code harder to maintain.

The normal transaction update process code is critical for general cases, it's not worthy to sacrifice the readability on behalf of non-general code like the temporary table.

(I'm not sure we can really achieve that goal ... just some personal thoughts.)

Yes, wrap it in the txn it self is more elegant but I wonder it need more works to archive it. For example for Iter method, to avoid send request to TiKV, we must split the range to multiple ranges and scan it foreach. I think we can create another pr to see how many modifications it needs.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 4, 2021
@tiancaiamao
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 34a7283

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 4, 2021
@ti-chi-bot
Copy link
Member

@lcwangchao: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@lcwangchao
Copy link
Collaborator Author

/run-check_dev_2

@ti-chi-bot ti-chi-bot merged commit 6a1e521 into pingcap:master Aug 4, 2021
@lcwangchao lcwangchao deleted the tmp_table_insert_ex branch October 18, 2021 02:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note-none Denotes a PR that doesn't merit a release note. sig/execution SIG execution sig/sql-infra SIG: SQL Infra size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Check local temporary table behavior 'insert ignore/on duplicate/replace'
4 participants