Skip to content
This repository has been archived by the owner on Dec 8, 2021. It is now read-only.

glue: add GlueCheckpointDB and remove external TiDB usage #478

Merged
merged 5 commits into from
Nov 23, 2020

Conversation

lance6716
Copy link
Contributor

@lance6716 lance6716 commented Nov 17, 2020

What problem does this PR solve?

complete glue to enable integrated into TiDB

What is changed and how it works?

  • add a glue implemented CheckpointDB, which is not elegant since it repeated many MySQLCheckpointDB code. Will refine later or in this PR.
  • resome some usage of TiDBManager. now NewTiDBManager is only called in tidb-lightning-ctl
  • remove usage of TiDB status port through TLS. now TLS is used only supposed to connect PD/TiKV
  • move database name from USE db to CREATE TABLE statements, because we may not rely on idle (sql.DB).ExecContext use same connection to execute sequentially SQLs, and in future we may support concurently create tables
  • remove taskID and replace it with cfg.TaskID, because taskID didn't have another value

Check List

Tests

  • manually test integrated into TiDB and import a sample data from integration test

Side effects

  • Increased code complexity

Related changes

add notes

save work

save work

fix unit test

remove tidbMgr in RestoreController

remove some comments

remove some comments

change logger in SQLWithRetry

revert replace log.Logger to *zap.Logger

dep: update uuid dependency to latest google/uuid (pingcap#452)

* dep: update satori/go.uuid to latest

* fix tests

* change to google/uuid

* fix build

* try fix test

* get familiar with google/uuid

* address comment

tidb-lightning-ctl: change default of -d to 'noop://' (pingcap#453)

also add noop:// to supported storage types (to represent an empty store)

replace tab to space

try another port to fix CI

remove some comment

*: more glue

restore: fix the bug that gc life time ttl does not take effect (pingcap#448)

* fix gc ttl loop

* resolve comment and add tests

fix CI

report info to host TiDB

config: filter out all system schemas by default (pingcap#459)

backend: fix auto random default value for primary key (pingcap#457)

* fix auto generate auto random primary key column

* fix default for auto random primary key

* fix test

* use prev row id for auto random and add a test

* replace chunck with session opt

* fix

* fix

mydumper: fix parquet data parser (pingcap#435)

* fix parquet

* reorder imports

* fix test

* use empty collation

* fix a error and add more test cases

* add pointer type tests

* resolve comments

Co-authored-by: kennytm <kennytm@gmail.com>

address comment

backend/local: use range properties to optimize region range estimate (pingcap#422)

* use range propreties to estimate region range

* post-restore: add optional level for post-restore operations (pingcap#421)

* add optional level for opst-restore operations

* trim leading and suffix '"

* use UnmarshalTOML to unmarshal post restore op level

* resolve comments and fix unit test

* backend/local: do not retry epochNotMatch error when ingest sst (pingcap#419)

* do not retry epochNotMatch error when ingest sst

* add retry ingest for 'Raft raft: proposal dropped' error in ingest

* change some retryable error log level from Error to Warn

* fix nextKey

* add a comment for nextKey

* fix comment and add a unit test

* wrap time.Sleep in select

Co-authored-by: kennytm <kennytm@gmail.com>

* update

* use range properties to optimze region range estimate

* update pebble

* change the default value for batch-size

* add unit tests and reslove comments

* add a comment to range properties test

* add a comment

* add a test for range property with pebble

* rename const variable

Co-authored-by: kennytm <kennytm@gmail.com>

fix pd service id is empty (pingcap#460)

fix s3 parquet reader (pingcap#461)

Co-authored-by: Neil Shen <overvenus@gmail.com>

fix service gc ttl again (pingcap#465)

address comment

mydumper: verify file routing config (pingcap#470)

* fix file routing

* remove useless line

* remove redundant if check

rename a method in interface

save work

try fix CI

could work

change ctx usage

try fix CI

try fix CI

refine function interface

refine some fucntion interface

debug CI

address comment

config: allow four byte-size config to be specified using human-readable units ("100 GiB") (pingcap#471)

* Makefile: add `make finish-prepare` action

* config: accept human-readable size for most byte-related config

e.g. allow `region-split-size = '96M'` in additional to `= 100663296`

(known issue: these values' precisions will be truncated to 53 bits
instead of supporting all 63 bits)

* restore: reduce chance of spurious errors from TestGcTTLManagerSingle

Co-authored-by: glorv <glorvs@163.com>

remove debug log

test: change double type syntax (pingcap#474)

address comment

checkpoint: add glue checkpoint

resolve cycle import

expose Retry

refine

change interface to cope with TiDB

fix SQL string

fix SQL

adjust interface to embedded in TiDB

could import now

reduce TLS

restore: add `glue.Glue` interface and other function (pingcap#456)

* save my work

* add notes

* save work

* save work

* fix unit test

* remove tidbMgr in RestoreController

* remove some comments

* remove some comments

* change logger in SQLWithRetry

* revert replace log.Logger to *zap.Logger

* replace tab to space

* try another port to fix CI

* remove some comment

* *: more glue

* report info to host TiDB

* fix CI

* address comment

* address comment

* rename a method in interface

* save work

* try fix CI

* could work

* change ctx usage

* try fix CI

* try fix CI

* refine function interface

* refine some fucntion interface

* debug CI

* address comment

* remove debug log

* address comment

modify code

add comment

refine some code
@lance6716
Copy link
Contributor Author

lance6716 commented Nov 17, 2020

To design an interface to implemented by both TiDB's data structure and database/sql, and make MySQLCheckpointDB reusable, I found those works are needed:

lines are significently less than glue_checkpoint.go.

I guess we must repeat some code of MySQLCheckpointDB (currently) or some code of go's SQL driver to implement glue checkpoint. Should I switch to another way? @kennytm @glorv @overvenus

@overvenus
Copy link
Member

For now, I don't think we need to implement 'Scan' and 'Exec'. We can go on with the current implementation.

Copy link
Collaborator

@kennytm kennytm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

lightning/checkpoints/checkpoints.go Outdated Show resolved Hide resolved
@lance6716 lance6716 added the status/DNM Do not merge, test is failing or blocked by another PR label Nov 20, 2020
@lance6716 lance6716 added status/PTAL This PR is ready for review. Add this label back after committing new changes and removed status/DNM Do not merge, test is failing or blocked by another PR labels Nov 20, 2020
@lance6716
Copy link
Contributor Author

lance6716 commented Nov 20, 2020

Add new change:

  • move database name from USE db to CREATE TABLE statements, because we may not rely on idle (sql.DB).ExecContext use same connection to execute sequentially SQLs, and in future we may support concurently create tables
  • remove taskID and replace it with cfg.TaskID, because taskID didn't have another value

PTAL @kennytm @glorv @overvenus

Copy link
Collaborator

@kennytm kennytm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kennytm kennytm added status/LGT1 One reviewer already commented LGTM (LGTM1) and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Nov 21, 2020
Copy link
Contributor

@glorv glorv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@glorv glorv added status/LGT2 Two reviewers already commented LGTM, ready for merge (LGTM2) and removed status/LGT1 One reviewer already commented LGTM (LGTM1) labels Nov 23, 2020
@glorv glorv merged commit da8b24e into pingcap:master Nov 23, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status/LGT2 Two reviewers already commented LGTM, ready for merge (LGTM2)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants