Add DB metrics: Cassandra #587

rinx · 2020-07-16T09:49:53Z

Signed-off-by: Rintaro Okamura rintaro.okamura@gmail.com

Description:

Add Cassandra metrics to these components
- meta
- backup
added metrics
- query execution total count
- query attempts total count
- query latency

Related Issue:

#343

How Has This Been Tested?:

nothing

Environment:

Go Version: 1.14.4
Docker Version: 19.03.8
Kubernetes Version: 1.18.2
NGT Version: 1.12.0

Types of changes:

Bug fix [type/bug]
New feature [type/feature]
Add tests [type/test]
Security related changes [type/security]
Add documents [type/documentation]
Refactoring [type/refactoring]
Update dependencies [type/dependency]
Update benchmarks and performances [type/bench]
Update CI [type/ci]

Changes to Core Features:

Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your core changes, as applicable?
Have you successfully ran tests with your changes locally?

Checklist:

I have read the CONTRIBUTING document.
I have checked open Pull Requests for the similar feature or fixes?
I have added tests and benchmarks to cover my changes.
I have ensured all new and existing tests passed.
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly.

pull-assistant · 2020-07-16T09:49:55Z

Best reviewed: commit by commit

Optimal code review plan (3 warnings)

✨ add cassandra metrics

...db/nosql/cassandra/observer.go 63% changes removed in ♻️ use int64 ...

✅ update

...ndra/service/cassandra_test.go 85% changes removed in ♻️ remove red...

♻️ modified metrics type and name

...b/nosql/cassandra/cassandra.go 75% changes removed in ♻️ use int64 ...

♻️ modified aggregation type

♻️ use int64 for ms total

♻️ fix

✅ update

♻️ remove redundant opts

✨ add query observer to backup manager cassandra

✨ add tags

♻️ revise

✨ add query attempts total

♻️ use same variable for metrics description

✅ update

Merge branch 'master' into feature/meta-cassandra/add-query-metrics

Powered by Pull Assistant. Last update e13c495 ... 8cc5893. Read the comment docs.

vdaas-ci · 2020-07-16T09:50:13Z

[CHATOPS:HELP] ChatOps commands.

🙆‍♀️ /approve - approve
💌 /changelog - add changelog comment
🍱 /format - format codes and add licenses
✅ /gen-test - generate test codes
🏷️ /label - add labels
⏪ /rebase - rebase master

internal/observability/metrics/db/nosql/cassandra/cassandra.go

codecov · 2020-07-17T01:13:57Z

Codecov Report

Merging #587 into master will increase coverage by 3.92%.
The diff coverage is 16.32%.

@@            Coverage Diff             @@
##           master     #587      +/-   ##
==========================================
+ Coverage    7.84%   11.77%   +3.92%     
==========================================
  Files         388      409      +21     
  Lines       19945    21302    +1357     
==========================================
+ Hits         1565     2508     +943     
- Misses      18156    18514     +358     
- Partials      224      280      +56

Impacted Files	Coverage Δ
internal/cache/cacher/cacher.go	`100.00% <ø> (+100.00%)`	⬆️
internal/client/discoverer/discover.go	`0.00% <0.00%> (ø)`
internal/compress/gob.go	`26.19% <0.00%> (-1.31%)`	⬇️
internal/compress/gzip.go	`26.98% <0.00%> (-7.02%)`	⬇️
internal/compress/lz4.go	`36.36% <0.00%> (-5.31%)`	⬇️
internal/compress/zstd.go	`29.50% <0.00%> (-5.79%)`	⬇️
internal/config/blob.go	`0.00% <0.00%> (ø)`
internal/config/config.go	`15.78% <ø> (+15.78%)`	⬆️
internal/config/grpc.go	`0.00% <0.00%> (ø)`
internal/config/log.go	`100.00% <ø> (+100.00%)`	⬆️
... and 172 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a5ac60c...8cc5893. Read the comment docs.

internal/observability/metrics/db/nosql/cassandra/cassandra.go

github-actions · 2020-07-20T02:51:29Z

internal/observability/metrics/db/nosql/cassandra/cassandra.go

+
+func (cm *cassandraMetrics) View() []*metrics.View {
+	return []*metrics.View{
+		&metrics.View{


[golangci] _{reported by reviewdog 🐶}
File is not gofmt-ed with -s (gofmt)

github-actions · 2020-07-21T08:35:05Z

internal/observability/metrics/mem/mem.go

@@ -55,7 +56,7 @@ func New() metrics.Metric {
 		heapReleased: *metrics.Int64(metrics.ValdOrg+"/memory/heap_released", "bytes of physical memory returned to the OS", metrics.UnitBytes),
 		stackInuse:   *metrics.Int64(metrics.ValdOrg+"/memory/stack_inuse", "bytes in stack spans", metrics.UnitBytes),
 		stackSys:     *metrics.Int64(metrics.ValdOrg+"/memory/stack_sys", "bytes of stack memory obtained from the OS", metrics.UnitBytes),
-		pauseTotalMs: *metrics.Float64(metrics.ValdOrg+"/memory/pause_ms_total", "the cumulative milliseconds in GC", metrics.UnitMilliseconds),
+		pauseTotalMs: *metrics.Int64(metrics.ValdOrg+"/memory/pause_ms_total", "the cumulative milliseconds in GC", metrics.UnitMilliseconds),


[golangci] _{reported by reviewdog 🐶}
line is 136 characters (lll)

github-actions · 2020-07-21T08:35:05Z

internal/observability/metrics/metrics.go

@@ -40,6 +40,8 @@ var (
 	LastValue    = view.LastValue
 	Sum          = view.Sum

+	DefaultMillisecondsDistribution = Distribution(0.01, 0.05, 0.1, 0.3, 0.6, 0.8, 1, 2, 3, 4, 5, 6, 8, 10, 13, 16, 20, 25, 30, 40, 50, 65, 80, 100, 130, 160, 200, 250, 300, 400, 500, 650, 800, 1000, 2000, 5000, 10000, 20000, 50000, 100000)


[golangci] _{reported by reviewdog 🐶}
line is 237 characters (lll)

github-actions · 2020-07-21T08:35:06Z

internal/observability/metrics/db/nosql/cassandra/cassandra.go

+
+func New() Observer {
+	return &cassandraMetrics{
+		queryTotal:   *metrics.Int64(metrics.ValdOrg+"/db/nosql/cassandra/completed_query_total", "cumulative count of completed queries", metrics.UnitDimensionless),


[golangci] _{reported by reviewdog 🐶}
line is 160 characters (lll)

github-actions · 2020-07-21T08:35:06Z

internal/observability/metrics/db/nosql/cassandra/cassandra.go

+func New() Observer {
+	return &cassandraMetrics{
+		queryTotal:   *metrics.Int64(metrics.ValdOrg+"/db/nosql/cassandra/completed_query_total", "cumulative count of completed queries", metrics.UnitDimensionless),
+		queryLatency: *metrics.Float64(metrics.ValdOrg+"/db/nosql/cassandra/query_latency", "query latency", metrics.UnitMilliseconds),


[golangci] _{reported by reviewdog 🐶}
line is 129 characters (lll)

github-actions · 2020-07-21T10:00:17Z

internal/observability/metrics/db/nosql/cassandra/cassandra.go

+	ms []metrics.MeasurementWithTags
+}
+
+type Observer interface {


[golangci] _{reported by reviewdog 🐶}
exported type Observer should have comment or be unexported (golint)

github-actions · 2020-07-21T10:00:17Z

internal/observability/metrics/db/nosql/cassandra/cassandra.go

+	cassandra.QueryObserver
+}
+
+func New() (o Observer, err error) {


[golangci] _{reported by reviewdog 🐶}
exported function New should have comment or be unexported (golint)

github-actions · 2020-07-21T10:00:17Z

internal/observability/metrics/db/nosql/cassandra/cassandra.go

+	cm.ms = append(
+		cm.ms,
+		metrics.MeasurementWithTags{
+			Measurement: cm.queryTotal.M(1),


[golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 1, in detected (gomnd)

github-actions · 2020-07-21T10:00:17Z

internal/db/nosql/cassandra/observer.go

+	"github.com/gocql/gocql"
+)
+
+type QueryObserver = gocql.QueryObserver


[golangci] _{reported by reviewdog 🐶}
exported type QueryObserver should have comment or be unexported (golint)

github-actions · 2020-07-21T10:00:17Z

internal/db/nosql/cassandra/observer.go

+)
+
+type QueryObserver = gocql.QueryObserver
+type ObservedQuery = gocql.ObservedQuery


[golangci] _{reported by reviewdog 🐶}
exported type ObservedQuery should have comment or be unexported (golint)

github-actions · 2020-07-21T10:00:18Z

internal/db/nosql/cassandra/observer.go

+type QueryObserver = gocql.QueryObserver
+type ObservedQuery = gocql.ObservedQuery
+
+type BatchObserver = gocql.BatchObserver


[golangci] _{reported by reviewdog 🐶}
exported type BatchObserver should have comment or be unexported (golint)

github-actions · 2020-07-21T10:00:18Z

internal/db/nosql/cassandra/observer.go

+
+type BatchObserver = gocql.BatchObserver
+
+type ConnectObserver = gocql.ConnectObserver


[golangci] _{reported by reviewdog 🐶}
exported type ConnectObserver should have comment or be unexported (golint)

github-actions · 2020-07-21T10:00:18Z

internal/db/nosql/cassandra/observer.go

+
+type ConnectObserver = gocql.ConnectObserver
+
+type FrameHeaderObserver = gocql.FrameHeaderObserver


[golangci] _{reported by reviewdog 🐶}
exported type FrameHeaderObserver should have comment or be unexported (golint)

internal/observability/metrics/db/nosql/cassandra/cassandra.go

rinx · 2020-07-27T11:37:49Z

/rebase
/format

vdaas-ci · 2020-07-27T11:38:26Z

[REBASE] Rebase triggered by rinx for branch: feature/meta-cassandra/add-query-metrics

vdaas-ci · 2020-07-27T11:41:42Z

[FORMAT] Updating license headers and formatting go codes triggered by rinx.

github-actions · 2020-07-27T11:44:04Z

internal/observability/metrics/metrics.go

@@ -40,6 +40,8 @@ var (
 	LastValue    = view.LastValue
 	Sum          = view.Sum

+	DefaultMillisecondsDistribution = Distribution(0.01, 0.05, 0.1, 0.3, 0.6, 0.8, 1, 2, 3, 4, 5, 6, 8, 10, 13, 16, 20, 25, 30, 40, 50, 65, 80, 100, 130, 160, 200, 250, 300, 400, 500, 650, 800, 1000, 2000, 5000, 10000, 20000, 50000, 100000)


[golangci] _{reported by reviewdog 🐶}
DefaultMillisecondsDistribution is a global variable (gochecknoglobals)

kpango

LGTM

rinx · 2020-07-28T14:46:29Z

/rebase

vdaas-ci · 2020-07-28T14:47:32Z

[REBASE] Rebase triggered by rinx for branch: feature/meta-cassandra/add-query-metrics

kpango · 2020-08-03T03:12:16Z

/rebase
/format
/approve

vdaas-ci · 2020-08-03T03:12:49Z

[REBASE] Rebase triggered by kpango for branch: feature/meta-cassandra/add-query-metrics

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

vdaas-ci · 2020-08-03T03:15:55Z

[FORMAT] Updating license headers and formatting go codes triggered by kpango.

vdaas-ci

[APPROVED] This PR is approved by kpango.

github-actions bot added area/internal area/meta size/L type/feature New feature priority/medium team/set SET team size/XL and removed size/L labels Jul 16, 2020

github-actions bot reviewed Jul 17, 2020

View reviewed changes

internal/observability/metrics/db/nosql/cassandra/cassandra.go Outdated Show resolved Hide resolved

github-actions bot reviewed Jul 20, 2020

View reviewed changes

rinx force-pushed the feature/meta-cassandra/add-query-metrics branch from 1255083 to c003660 Compare July 21, 2020 03:11

github-actions bot reviewed Jul 21, 2020

View reviewed changes

github-actions bot added the area/manager/backup label Jul 21, 2020

github-actions bot reviewed Jul 21, 2020

View reviewed changes

rinx force-pushed the feature/meta-cassandra/add-query-metrics branch from 438ed70 to 39d4911 Compare July 21, 2020 10:17

github-actions bot reviewed Jul 22, 2020

View reviewed changes

internal/observability/metrics/db/nosql/cassandra/cassandra.go Show resolved Hide resolved

rinx marked this pull request as ready for review July 22, 2020 03:34

vdaas-ci force-pushed the feature/meta-cassandra/add-query-metrics branch from 4776087 to 07434c0 Compare July 27, 2020 11:38

github-actions bot reviewed Jul 27, 2020

View reviewed changes

rinx requested review from kpango and datelier July 27, 2020 12:46

kpango previously approved these changes Jul 27, 2020

View reviewed changes

datelier previously approved these changes Jul 28, 2020

View reviewed changes

vdaas-ci dismissed stale reviews from datelier and kpango via 2513c9c July 28, 2020 14:47

vdaas-ci force-pushed the feature/meta-cassandra/add-query-metrics branch from 07434c0 to 2513c9c Compare July 28, 2020 14:47

rinx added 14 commits August 3, 2020 03:12

✨ add cassandra metrics

e13c495

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✅ update

f5b96bd

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ modified metrics type and name

11baf52

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ modified aggregation type

9274213

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ use int64 for ms total

08deb26

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ fix

34a200d

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✅ update

f6fbeaf

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ remove redundant opts

7acd4a0

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✨ add query observer to backup manager cassandra

34d7158

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✨ add tags

ec74eb4

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ revise

409882f

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✨ add query attempts total

c06e83e

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ use same variable for metrics description

9dd64b6

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✅ update

b95d636

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

vdaas-ci force-pushed the feature/meta-cassandra/add-query-metrics branch from 2513c9c to b95d636 Compare August 3, 2020 03:12

vdaas-ci approved these changes Aug 3, 2020

View reviewed changes

Merge branch 'master' into feature/meta-cassandra/add-query-metrics

8cc5893

kpango merged commit 2348bba into master Aug 3, 2020

kpango deleted the feature/meta-cassandra/add-query-metrics branch August 3, 2020 03:26

vdaas-ci mentioned this pull request Aug 3, 2020

[patch] Add warn logging messages to agent-sidecar & ignore io.EOF error when reading metadata.json #608

Merged

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DB metrics: Cassandra #587

Add DB metrics: Cassandra #587

rinx commented Jul 16, 2020 •

edited

Loading

pull-assistant bot commented Jul 16, 2020 •

edited

Loading

vdaas-ci commented Jul 16, 2020

codecov bot commented Jul 17, 2020 •

edited

Loading

github-actions bot Jul 20, 2020

github-actions bot Jul 21, 2020

github-actions bot Jul 21, 2020

github-actions bot Jul 21, 2020

github-actions bot Jul 21, 2020

github-actions bot Jul 21, 2020

github-actions bot Jul 21, 2020

github-actions bot Jul 21, 2020

github-actions bot Jul 21, 2020

github-actions bot Jul 21, 2020

github-actions bot Jul 21, 2020

github-actions bot Jul 21, 2020

github-actions bot Jul 21, 2020

rinx commented Jul 27, 2020

vdaas-ci commented Jul 27, 2020

vdaas-ci commented Jul 27, 2020

github-actions bot Jul 27, 2020

kpango left a comment

rinx commented Jul 28, 2020

vdaas-ci commented Jul 28, 2020

kpango commented Aug 3, 2020

vdaas-ci commented Aug 3, 2020

vdaas-ci commented Aug 3, 2020

vdaas-ci left a comment


		type BatchObserver = gocql.BatchObserver

		type ConnectObserver = gocql.ConnectObserver


		type ConnectObserver = gocql.ConnectObserver

		type FrameHeaderObserver = gocql.FrameHeaderObserver

Add DB metrics: Cassandra #587

Add DB metrics: Cassandra #587

Conversation

rinx commented Jul 16, 2020 • edited Loading

Description:

Related Issue:

How Has This Been Tested?:

Environment:

Types of changes:

Changes to Core Features:

Checklist:

pull-assistant bot commented Jul 16, 2020 • edited Loading

vdaas-ci commented Jul 16, 2020

codecov bot commented Jul 17, 2020 • edited Loading

Codecov Report

github-actions bot Jul 20, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

github-actions bot Jul 21, 2020

Choose a reason for hiding this comment

rinx commented Jul 27, 2020

vdaas-ci commented Jul 27, 2020

vdaas-ci commented Jul 27, 2020

github-actions bot Jul 27, 2020

Choose a reason for hiding this comment

kpango left a comment

Choose a reason for hiding this comment

rinx commented Jul 28, 2020

vdaas-ci commented Jul 28, 2020

kpango commented Aug 3, 2020

vdaas-ci commented Aug 3, 2020

vdaas-ci commented Aug 3, 2020

vdaas-ci left a comment

Choose a reason for hiding this comment

rinx commented Jul 16, 2020 •

edited

Loading

pull-assistant bot commented Jul 16, 2020 •

edited

Loading

codecov bot commented Jul 17, 2020 •

edited

Loading