New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make simulator Great Again! #8135

Open

HuSharp opened this issue Apr 29, 2024 · 0 comments

Assignees

Labels

type/development

Member

HuSharp commented Apr 29, 2024 •

edited

Loading

Development Task

Currently, we have many components to simulate the interaction with pd, pd-api-bench, pd-tso-bench, pd-heartbeat-bench, pd-simulator. We can integrate them step by step, with lower learning costs and easier interaction testing of various pd's.
- Enhance simulator to support 1000w regions and 300 stores
- Enhance heartbeat bench for daily regression testing
  - tools/heartbeat: support to collect metrics #8235
- Enhance tso bench
  - tools/tso: add option useTSOServerProxy in pd-tso-bench #7756
After that we can provide pd the same specification machine to deploy the real pd under the big cluster, simulate the request number of TiDB/TiKV + TiKV heartbeat heartbeat report + all kinds of scheduling for the region and so on.
- tools: support pd dev #8140

mock TiKV data to archive the following goals:

can balance region...
can support three forms node: meta-node/simulator(simple raft)/real-tikv
support unified interface to interact with real-tikv

mock TiDB&TiKV API

existed tool usages like

./bin/pd-api-bench -http-cases GetRegionStatus-1+1,GetMinResolvedTS-1+1 -client 1 -debug true

mock TiKV heartbeat/TSO

existed tool usages like
./bin/pd-heartbeat-bench --config=./tools/pd-heartbeat-bench/config-template.toml --pd-endpoints http://127.0.0.1:2379

Others

support front operator

Related

integration simulator tools tools: support pd dev #8140

#5468 extend simulator
#6922 extend API bench

The text was updated successfully, but these errors were encountered:

HuSharp added the type/development label

HuSharp self-assigned this

HuSharp mentioned this issue

tools: support pd dev #8140

Draft

This was referenced May 27, 2024

simulator: make store,region,replica configurable in cases #8215

Merged

simulator: scale-in and scale out test for store-limit-version=v2 #8220

Merged

ti-chi-bot bot added a commit that referenced this issue


          simulator: make store,region,replica configurable in cases (#8215)

52389b0

ref #8135

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

This was referenced May 31, 2024

tools/heartbeat: support to collect metrics #8235

Merged

tools/simulator: add store api and replace simulator http with SDK #8245

Merged

tools/simulator: support deleting a specified store #8246

Merged

ti-chi-bot bot added a commit that referenced this issue


          tools/simulator: add store api and replace simulator http with SDK (#…

301fabb

…8245)

ref #8135

Signed-off-by: husharp <jinhao.hu@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

HuSharp mentioned this issue

tools/simulator: avoid redundant schedule #8257

Merged

ti-chi-bot bot added a commit that referenced this issue


          tools/simulator: avoid redundant schedule (#8257)

494c0e9

close #5290, ref #8135

Signed-off-by: husharp <jinhao.hu@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

HuSharp mentioned this issue

tools/simulator: Make simulator work with large scale cluster #8269

Merged

HuSharp mentioned this issue

tools/simulator: support simulator with multiple pds #8304

Merged

ti-chi-bot bot added a commit that referenced this issue


          tools/simulator: support simulator with multiple pds (#8304)

89ca8df

ref #8135

Signed-off-by: husharp <jinhao.hu@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

ti-chi-bot bot added a commit that referenced this issue


          tools/simulator: support deleting a specified store (#8246)

41e6376

ref #8135

Signed-off-by: husharp <jinhao.hu@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

This was referenced Jun 21, 2024

tools/simulator: reduce range times #8318

Merged

tools/tso: add option useTSOServerProxy in pd-tso-bench #7756

Merged

ti-chi-bot bot added a commit that referenced this issue


          tools/tso: add option useTSOServerProxy in pd-tso-bench (#7756)

8215f30

ref #8135

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: Hu# <jinhao.hu@pingcap.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

ti-chi-bot bot added a commit that referenced this issue


          tools/simulator: reduce range times (#8318)

6ae7b47

ref #8135

Signed-off-by: husharp <jinhao.hu@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

HuSharp mentioned this issue

tools/simulator: replace GenerateSplitKey with GenerateTableKeys #8332

Merged

ti-chi-bot bot pushed a commit that referenced this issue


          tools/simulator: replace GenerateSplitKey with GenerateTableKeys (#8332)

ab7f903

ref #8135

Signed-off-by: husharp <jinhao.hu@pingcap.com>

ti-chi-bot bot pushed a commit that referenced this issue


          simulator: scale-in and scale out test for store-limit-version=v2 (#…

e99b3ac

…8220)

ref #8135

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 mentioned this issue

simulator: fix region size and keys #8366

Merged

ti-chi-bot bot pushed a commit that referenced this issue


          simulator: fix region size and keys (#8366)

aecb4ab

ref #8135

Signed-off-by: lhy1024 <admin@liudos.us>

HuSharp mentioned this issue

tools/simulator: resolve simulator race #8376

Merged

ti-chi-bot bot added a commit that referenced this issue


          tools/simulator: resolve simulator race (#8376)

90273dd

ref #8135

Signed-off-by: husharp <ihusharp@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

ti-chi-bot bot added a commit that referenced this issue


          tools/simulator: Make simulator work with large scale cluster (#8269)

2baee83

close #5683, ref #8135

Signed-off-by: husharp <jinhao.hu@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

ti-chi-bot bot added a commit that referenced this issue


          tools/heartbeat: support to collect metrics (#8235)

5d77447

ref #8135

tools/heartbeat: support to collect metrics

Signed-off-by: husharp <jinhao.hu@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment