tikvclient: add metrics for gRPC connection transient failure #12084
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #12084 +/- ##
===========================================
Coverage 82.1858% 82.1858%
===========================================
Files 447 447
Lines 99567 99567
===========================================
Hits 81830 81830
Misses 12148 12148
Partials 5589 5589 |
|
/bench |
lysu
reviewed
Sep 9, 2019
|
/run-all-tests |
@@ Benchmark Diff @@
================================================================================
--- tidb: 619b9a92b3c26b830687ea06ffe25538c464505d
+++ tidb: 893f2c93ef8154906765e7b96ddc1c075a16c7a0
tikv: b4daac2bb80ade14d1442d6fc21fa9c12efab943
pd: b66ba4482c5dfb3d976461544b5df3b8442d4d37
================================================================================
test-1: < oltp_insert >
* QPS : 21367.25 ± 0.7889% (std=102.98) delta: -0.14%
* AvgMs : 11.98 ± 0.8016% (std=0.06) delta: -0.10%
* PercentileMs99 : 42.61 ± 0.0000% (std=0.00) delta: -0.37%
test-2: < oltp_update_non_index >
* QPS : 29570.68 ± 0.5176% (std=90.24) delta: 0.31%
* AvgMs : 8.64 ± 0.2315% (std=0.01) delta: -0.46%
* PercentileMs99 : 30.59 ± 2.5431% (std=0.44) delta: -0.36%
test-3: < oltp_read_write >
* QPS : 36750.93 ± 0.1727% (std=56.07) delta: -0.26%
* AvgMs : 139.86 ± 0.1698% (std=0.20) delta: 0.26%
* PercentileMs99 : 257.95 ± 0.0000% (std=0.00) delta: 0.00%
test-4: < oltp_point_select >
* QPS : 76150.13 ± 0.7291% (std=322.87) delta: -0.69%
* AvgMs : 3.37 ± 0.3712% (std=0.01) delta: 0.90%
* PercentileMs99 : 7.43 ± 0.0000% (std=0.00) delta: 0.00%
test-5: < oltp_update_index >
* QPS : 16981.31 ± 0.4707% (std=62.66) delta: -0.16%
* AvgMs : 15.07 ± 0.4512% (std=0.05) delta: 0.16%
* PercentileMs99 : 47.99 ± 1.0877% (std=0.43) delta: -0.72%
|
lonng
changed the title
Signed-off-by: Lonng <heng@lonng.org>
disksing
approved these changes
Sep 9, 2019
@@ Benchmark Diff @@
================================================================================
--- tidb: 619b9a92b3c26b830687ea06ffe25538c464505d
+++ tidb: 893f2c93ef8154906765e7b96ddc1c075a16c7a0
tikv: b4daac2bb80ade14d1442d6fc21fa9c12efab943
pd: b66ba4482c5dfb3d976461544b5df3b8442d4d37
================================================================================
tidb_max_cpu: 19.21, delta: -0.46%
tikv_max_cpu: 13.72, delta: 3.66%
tidb_max_memory: 1718.83 MiB, delta: -3.11%
tikv_max_memory: 59880.01 MiB, delta: 0.03%
Measured tpmC (NewOrders): 21651.64, delta: -2.49% |
|
/run-all-tests |
|
/run-cherry-picker |
|
cherry pick to release-3.0 failed |
|
cherry pick to release-2.1 failed |
This was referenced Sep 9, 2019
sre-bot
pushed a commit
that referenced
this pull request
Sep 9, 2019
sre-bot
pushed a commit
that referenced
this pull request
Sep 9, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Lonng heng@lonng.org
What problem does this PR solve?
TiDB access to TiKV through gRPC requests. If the underlying socket is disconnected, gRPC will try to reconnect to the underlying socket, which may cause the request delay to jitter. We need to way to monitor the low-level socket state change.
What is changed and how it works?
This PR adds a metric to monitor the gRPC connection state, the metric will record the connection state before sending the request to TiKV. We can diagnose the delay jitter by
rate(tidb_grpc_connection_state)after this PR merged.Check List
Tests
Related changes
Release note