-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vttablet throttling #6668
vttablet throttling #6668
Conversation
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
- tracking `lastCheckTimeNano` - isDormant(): infrequent sampling when no app runs checks - upon transitioning to leader (table type is `MASTER`), create a MySQL account with random password and reconfigure throttler to use that password - ensure to re-attempt user creation until successful Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
- tracking `lastCheckTimeNano` - isDormant(): infrequent sampling when no app runs checks - upon transitioning to leader (table type is `MASTER`), create a MySQL account with random password and reconfigure throttler to use that password - ensure to re-attempt user creation until successful Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
This is now pretty much feature complete and ready to be used (notwithstanding endtoend tests) General behavior description:
|
Question: how should the throttler respond if there are no $ curl -s http://sn-carbon:15101/throttler/check | jq . {
"StatusCode":500,
"Value":0,
"Threshold":1,
"Message":"No hosts found"
} should we instead return a |
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
endtoend tests are good. |
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
API examples: on MASTER tablet$ curl -s http://sn-carbon:15100/throttler/status | jq . {
"Keyspace": "commerce",
"Shard": "0",
"IsLeader": true,
"IsOpen": true,
"IsDormant": false,
"AggregatedMetrics": {
"mysql/local": {
"Value": 0.074892
}
},
"MetricsHealth": {}
} curl -s http://sn-carbon:15100/throttler/check | jq -c {"StatusCode":200,"Value":0.107066,"Threshold":1,"Message":""} $ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global StopReplication zone1-0000000101
$ curl -s http://sn-carbon:15100/throttler/check | jq -c {"StatusCode":429,"Value":3.14063,"Threshold":1,"Message":"Threshold exceeded"} vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global ChangeTabletType zone1-0000000101 RDONLY
# wait...
$ curl -s http://sn-carbon:15100/throttler/check | jq -c
{"StatusCode":500,"Value":0,"Threshold":1,"Message":"No hosts found"} $ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global StartReplication zone1-0000000101
$ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global ChangeTabletType zone1-0000000101 REPLICA
# wait...
$ curl -s http://sn-carbon:15100/throttler/check | jq -c {"StatusCode":200,"Value":0.113081,"Threshold":1,"Message":""} on REPLICA tabletOnly the $ curl -s http://sn-carbon:15101/throttler/status | jq . {
"Keyspace": "commerce",
"Shard": "0",
"IsLeader": false,
"IsOpen": true,
"IsDormant": true,
"AggregatedMetrics": {},
"MetricsHealth": {}
} $ curl -s http://sn-carbon:15101/throttler/check | jq -c {"StatusCode":404,"Value":0,"Threshold":0,"Message":"No such metric"} |
This PR assumes |
…ing a bug where CREATE USER hangs on MySQL 8.0.21 Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Code refactored for different behavior on |
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
d43937f
to
b9d4314
Compare
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
ping, request for review 🙏 |
Per @sougou , if there's no replicas, the throttler should report |
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
OK, updated the code to return |
Just a drive-by observation: seems like it might be worthwhile to be able to configure whether Many of us are running with only For those of us with only |
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Introduced
it's a comma delimited list of tablet types which the throttler checks for lag. Tablet types not in this list are completely ignored. This answers @dweitzman 's comment |
assert.Equal(t, http.StatusNotFound, resp.StatusCode) | ||
} | ||
|
||
func TestThrottlerAfterMetricsCollected(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does it make sense for users to configure the polling interval?
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Throttler now enables heartbeat writer, regardless of |
Context: #6661
This PR introduces a vttablet throttler. Based on https://github.com/github/freno, and adapted to vitess, a vttablet throttler runs on a
MASTER
tablet, and probes and serves throttling on its own cluster (ie keyspace+shard). The throttler only considersREPLICA
servers in the topology (ie it skipsRDONLY
).A throttler also runs on replicas, but does nothing, until such time that the replica becomes a
MASTER
.The logic as imported from
freno
is more general purpose. Large parts of the code were stripped out. Some generalization still exists (e.g. the notion of multiple clusters, even though a throttler in this PR only handles a single cluster).The code is tightly integrated with vitess: leadership determined by
Tablet.Type
; detection ofREPLICA
servers done directly viatopo.Server
, etc.Ongoing work:
pt-online-schema-change
,gh-ost
)_vt.heartbeat
: