Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support TiKV auto scaling-out by Storage #2884

Merged
merged 17 commits into from
Jul 13, 2020
Merged

Support TiKV auto scaling-out by Storage #2884

merged 17 commits into from
Jul 13, 2020

Conversation

Yisaer
Copy link
Contributor

@Yisaer Yisaer commented Jul 8, 2020 •
edited
Loading

What problem does this PR solve?

close #2909

Support auto scaling-out TiKV by Storage. The Storage Auto-scaling Strategy is similar to the AWS RDS Storage Auto-scaling

The tikv would be auto-scaling out in these rules:

  1. If the available storage size have already been less than the 10%(could be customized) of the capacity storage size, the cluster status will be marked as Storage Pressure.
  2. If the status Storage Pressure has been existed for 5min(could be customized), the tikv would be scaled-out.

Note that the Max-Replicas and ScaleOut Cool-down is still worked in this case.

Note that in v1.1.x, auto-scaling by storage and cpu couldn't be both work for tikv. We will support this feature in v1.2.x

If the rule based by cpu and based by storage are both set, only rule based by storage would be considered.

Here is one example for the tidbclusterautoscaler to create auto-scaling by storage :

apiVersion: pingcap.com/v1alpha1
kind: TidbClusterAutoScaler
metadata:
  name: basic
spec:
  cluster:
    name: basic
    namespace: tidb-admin
  monitor:
    name: monitor
    namespace: tidb-admin
  tikv:
    maxReplicas: 3
    scaleOutIntervalSeconds: 500
    metrics:
      - type: "Resource"
        leastStoragePressurePeriodSeconds: 300
        leastRemainAvailableStoragePercent: 10
        resource:
          name: "storage"

The Status would be like below (normal):

status:
  tikv:
    currentReplicas: 1
    lastAutoScalingTimestamp: "2020-07-08T11:30:38Z"
    metrics:
    - availableStorage: 100.1 GB
      baselineAvailableStorage: 15.6 GB
      capacityStorage: 156.4 GB
      name: storage
      storagePressure: false

During Storage Pressure:

status:
  tikv:
    currentReplicas: 1
    lastAutoScalingTimestamp: "2020-07-08T11:33:44Z"
    metrics:
    - availableStorage: 100.1 GB
      baselineAvailableStorage: 140.8 GB
      capacityStorage: 156.4 GB
      name: storage
      storagePressure: true
      storagePressureStartTime: "2020-07-08T11:33:44Z"

What is changed and how does it work?

Check List

Tests

  • Unit test
  • E2E test

Related changes

  • Need to cherry-pick to the release branch

Does this PR introduce a user-facing change?:

Support Auto-scaling by Storage for TiKV in TidbClusterAutoScaler

@Yisaer Yisaer marked this pull request as draft July 8, 2020 07:11
@Yisaer Yisaer changed the title Support TiKV Auto-scaling by Storage Support TiKV auto scaling-out by Storage Jul 8, 2020
@Yisaer
add storage api
75ead5d 
fix

fix

fix

Update tikv_autoscaler.go

fix

fix

fix lint

add doc

fix error

fix error

fix unit test

fix error

remove useless code

fux unit test

fix error

Update docs.md

fix api

fix status
@Yisaer Yisaer marked this pull request as ready for review July 8, 2020 12:39
@Yisaer Yisaer added the status/PTAL PR needs to be reviewed label Jul 8, 2020
@Yisaer
Copy link
Contributor Author

Yisaer commented Jul 9, 2020

/pull-e2e-kind-serial

1 similar comment
@Yisaer
Copy link
Contributor Author

Yisaer commented Jul 9, 2020

/pull-e2e-kind-serial

@Yisaer

This comment has been minimized.

Sign in to view
@Yisaer

This comment has been minimized.

Sign in to view
2 similar comments
@Yisaer
Copy link
Contributor Author

Yisaer commented Jul 9, 2020

/test pull-e2e-kind-serial

@Yisaer
Copy link
Contributor Author

Yisaer commented Jul 9, 2020

/test pull-e2e-kind-serial

@cofyc
Copy link
Contributor

cofyc commented Jul 13, 2020

I would suggest using the phrase storage-based auto-scaling instead of auto-scaling by storage:

Support storage-based auto-scaling for TiKV in TidbClusterAutoScaler

cofyc
cofyc previously approved these changes Jul 13, 2020
View reviewed changes
@ti-srebot
Copy link
Contributor

@cofyc,Thanks for your review.

@Yisaer Yisaer requested a review from cofyc July 13, 2020 04:58
@ti-srebot
Copy link
Contributor

@weekface,Thanks for your review.

@Yisaer
Copy link
Contributor Author

Yisaer commented Jul 13, 2020

/merge

@ti-srebot
Copy link
Contributor

/run-all-tests

1 similar comment
@Yisaer
Copy link
Contributor Author

Yisaer commented Jul 13, 2020

/run-all-tests

@Yisaer

This comment has been minimized.

Sign in to view
3 similar comments
@Yisaer

This comment has been minimized.

Sign in to view
@Yisaer

This comment has been minimized.

Sign in to view
@Yisaer

This comment has been minimized.

Sign in to view
@Yisaer

This comment has been minimized.

Sign in to view
@Yisaer
Copy link
Contributor Author

Yisaer commented Jul 13, 2020

/test pull-e2e-kind-asts

@Yisaer Yisaer merged commit e0a5137 into pingcap:master Jul 13, 2020
ti-srebot pushed a commit to ti-srebot/tidb-operator that referenced this pull request Jul 13, 2020
@Yisaer @ti-srebot
cherry pick pingcap#2884 to release-1.1
fd0fbf8 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Jul 13, 2020
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-1.1 in PR #2917

Yisaer added a commit that referenced this pull request Jul 14, 2020
@ti-srebot @Yisaer
cherry pick #2884 to release-1.1 (#2917)
d48cc1e 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: Song Gao <disxiaofei@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ti-srebot ti-srebot ti-srebot approved these changes

@cofyc cofyc cofyc approved these changes

@weekface weekface weekface approved these changes

@zjj2wry zjj2wry Awaiting requested review from zjj2wry

@lichunzhu lichunzhu Awaiting requested review from lichunzhu

@DanielZhangQD DanielZhangQD Awaiting requested review from DanielZhangQD

Assignees
No one assigned
Labels
area/auto-scaling related to auto-scaling CanMerge needs-cherry-pick-1.1 status/LGT2 status/PTAL PR needs to be reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support Storage Auto-scaling for TiKV
5 participants
@Yisaer @cofyc @ti-srebot @weekface @DanielZhangQD