Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

metrics: improve reading Go runtime metrics #25886

Merged
merged 30 commits into from
Nov 11, 2022
Merged

metrics: improve reading Go runtime metrics #25886

merged 30 commits into from
Nov 11, 2022

Conversation

fjl
Copy link
Contributor

@fjl fjl commented Sep 28, 2022 •
edited
Loading

This PR changes how we read performance metrics from the Go runtime. Instead of
using runtime.ReadMemStats, we now rely on the API provided by package runtime/metrics.

Here's why:

ReadMemStats is a stop-the-world operation, It requires halting all goroutines,
resulting in very unpredictable latency. Under high load, it can sometimes take milliseconds
to run, and all other work will be interrupted while it waits.

runtime/metrics also provides more accurate information. For example, the new interface
has better reporting of memory use. In my testing, the reported value of held memory
more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of allocations
in number-of-objects. This is imprecise: 'tiny objects' are not counted because the runtime
allocates them in batches; and certain improvements in allocation behavior, such as struct
size optimizations, will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes will make it appear in graphs that lots more is
being allocated. I don't think that's a problem because this metric is primarily interesting
for geth developers.

I have also deleted a lot of old and unused code for reading runtime metrics. The functions
in metrics/runtime*.go were never used in go-ethereum.

@fjl
Copy link
Contributor Author

fjl commented Sep 28, 2022

GC pause metric doesn't work yet

@fjl fjl removed the status:triage label Oct 12, 2022
@fjl fjl force-pushed the metrics-runtime-metrics branch 2 times, most recently from 24dff68 to 825acd1 Compare November 9, 2022 21:09
fjl added 25 commits November 9, 2022 22:12
@fjl
metrics: WIP use runtime/metrics
445187c
@fjl
metrics: fixup
41cc612
@fjl
metrics: use gauge for gc metrics
993a51b
@fjl
metrics: update
516f5a9
@fjl
metrics: update
1a4bbce
@fjl
metrics: improve loop
7c1a4ac
@fjl
metrics: track alloc/free in objects not gc
41706c8
@fjl
metrics: rename variables
c0b8c7b
@fjl
metrics: fixup
3705d41
@fjl
metrics: track GC alloc/free in bytes
69e9bbf 
The bytes-based metric is better because it includes 'tiny' objects.
It's also easier to understand.
@fjl
metrics: skip invalid/unknown Go runtime metrics
182856d
@fjl
metrics: explain why skipped
87ca27d
@fjl
metrics: change GC pauses back to meter
58194f5
@fjl
metrics: add Float64Histogram adapter
432760b
@fjl
metrics: fix some issues in runtimeHistogram
32a49ce
@fjl
metrics: add tests
42afa68
@fjl
metrics: handle nil histogram updates
e223b03
@fjl
metrics: add some more runtime metrics
d5142b0
@fjl
metrics: improve handling of +Inf value in bucket
a8dd515 
The result of Histogram methods should never be Inf or NaN because the
metrics exporters can't handle these values.
@fjl
metrics: actually read new runtime metrics
e0712aa
@fjl
metrics: add scale factor
4419d87
@fjl
metrics: add +Inf workaround in Max
0db04a1
@fjl
metrics/influxdb: add .25 percentile in histogram output
825acd1
@fjl
metrics: remove allocs-bysize
3503c11
@fjl
metrics: add more heap metrics
152df4e
holiman
holiman approved these changes Nov 10, 2022
View reviewed changes
Copy link
Contributor

@holiman holiman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I think we should merge it and then try it out on our infra.

@fjl fjl merged commit c539bda into ethereum:master Nov 11, 2022
@fjl fjl added this to the 1.11.0 milestone Nov 11, 2022
shekhirin pushed a commit to shekhirin/go-ethereum that referenced this pull request Jun 6, 2023
@fjl @shekhirin
metrics: improve reading Go runtime metrics (ethereum#25886)
632443e 
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.

runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.

The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.

We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
@gzliudan gzliudan mentioned this pull request Dec 10, 2024
Merged
19 tasks
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Dec 10, 2024
@gzliudan
metrics: improve reading Go runtime metrics (ethereum#25886)
a8fe675 
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.

runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.

The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.

We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Dec 10, 2024
@gzliudan
metrics: improve reading Go runtime metrics (ethereum#25886)
d6bf161 
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.

runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.

The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.

We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Dec 10, 2024
@gzliudan
metrics: improve reading Go runtime metrics (ethereum#25886)
2864041 
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.

runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.

The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.

We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Dec 13, 2024
@gzliudan
metrics: improve reading Go runtime metrics (ethereum#25886)
dddd6c5 
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.

runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.

The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.

We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
JukLee0ira pushed a commit to JukLee0ira/XDPoSChain that referenced this pull request Dec 16, 2024
@gzliudan @JukLee0ira
metrics: improve reading Go runtime metrics (ethereum#25886)
2d9b6f2 
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.

runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.

The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.

We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
JukLee0ira pushed a commit to JukLee0ira/XDPoSChain that referenced this pull request Dec 16, 2024
@gzliudan @JukLee0ira
metrics: improve reading Go runtime metrics (ethereum#25886)
4e27af2 
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.

runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.

The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.

We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
JukLee0ira pushed a commit to JukLee0ira/XDPoSChain that referenced this pull request Dec 20, 2024
@gzliudan @JukLee0ira
metrics: improve reading Go runtime metrics (ethereum#25886)
122701e 
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.

runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.

The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.

We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
JukLee0ira pushed a commit to JukLee0ira/XDPoSChain that referenced this pull request Dec 20, 2024
@gzliudan @JukLee0ira
metrics: improve reading Go runtime metrics (ethereum#25886)
14243b9 
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.

runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.

The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.

We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
JukLee0ira pushed a commit to JukLee0ira/XDPoSChain that referenced this pull request Dec 22, 2024
@gzliudan @JukLee0ira
metrics: improve reading Go runtime metrics (ethereum#25886)
16df404 
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.

runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.

The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.

We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
JukLee0ira pushed a commit to JukLee0ira/XDPoSChain that referenced this pull request Jan 2, 2025
@gzliudan @JukLee0ira
metrics: improve reading Go runtime metrics (ethereum#25886)
5d6d2aa 
This changes how we read performance metrics from the Go runtime. Instead
of using runtime.ReadMemStats, we now rely on the API provided by package
runtime/metrics.

runtime/metrics provides more accurate information. For example, the new
interface has better reporting of memory use. In my testing, the reported
value of held memory more accurately reflects the usage reported by the OS.

The semantics of metrics system/memory/allocs and system/memory/frees have
changed to report amounts in bytes. ReadMemStats only reported the count of
allocations in number-of-objects. This is imprecise: 'tiny objects' are not
counted because the runtime allocates them in batches; and certain
improvements in allocation behavior, such as struct size optimizations,
will be less visible when the number of allocs doesn't change.

Changing allocation reports to be in bytes makes it appear in graphs that
lots more is being allocated. I don't think that's a problem because this
metric is primarily interesting for geth developers.

The metric system/memory/pauses has been changed to report statistical
values from the histogram provided by the runtime. Its name in influxdb has
changed from geth.system/memory/pauses.meter to
geth.system/memory/pauses.histogram.

We also have a new histogram metric, system/cpu/schedlatency, reporting the
Go scheduler latency.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@holiman holiman holiman approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.11.0
Development

Successfully merging this pull request may close these issues.
3 participants
@fjl @holiman @ligi