Add cfs-throttling example #311
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Example throttled program:
```
ivan@vm:~$ sudo systemd-run --pty --quiet --collect --unit stress-ng.service --property CPUQuota=8% stress-ng --cpu 1
stress-ng: info: [70953] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info: [70953] dispatching hogs: 1 cpu
```
One can observe throttling with `bpftrace`:
```
ivan@vm:~/projects/ebpf_exporter$ sudo bpftrace -e 'kprobe:unthrottle_cfs_rq { $cfs_rq = (struct cfs_rq *) arg0; $throttled_nsec = $cfs_rq->rq->clock - $cfs_rq->throttled_clock; printf("unthrottle after %4dms (overall: %6lums): %s\n", $throttled_nsec / 1000000, $cfs_rq->tg->cfs_bandwidth.throttled_time / 1000000, cgroup_path($cfs_rq->tg->css.cgroup->kn->id)); }'
Attaching 1 probe...
unthrottle after 95ms (overall: 520871ms): unified:/system.slice/stress-ng.service
unthrottle after 91ms (overall: 520966ms): unified:/system.slice/stress-ng.service
unthrottle after 92ms (overall: 521058ms): unified:/system.slice/stress-ng.service
unthrottle after 88ms (overall: 521150ms): unified:/system.slice/stress-ng.service
unthrottle after 91ms (overall: 521239ms): unified:/system.slice/stress-ng.service
unthrottle after 96ms (overall: 521330ms): unified:/system.slice/stress-ng.service
unthrottle after 87ms (overall: 521426ms): unified:/system.slice/stress-ng.service
unthrottle after 95ms (overall: 521514ms): unified:/system.slice/stress-ng.service
unthrottle after 90ms (overall: 521610ms): unified:/system.slice/stress-ng.service
unthrottle after 90ms (overall: 521700ms): unified:/system.slice/stress-ng.service
```
The values here also match what `cpu.stat` is reporting:
```
ivan@vm:~/projects/ebpf_exporter$ cat /sys/fs/cgroup/system.slice/stress-ng.service/cpu.stat
usage_usec 45447035
user_usec 45378526
system_usec 68508
nr_periods 5681
nr_throttled 5680
throttled_usec 522346085
nr_bursts 0
burst_usec 0
```
Example output from `ebpf_exporter`:
```
ivan@vm:~/projects/ebpf_exporter$ curl -s http://ip6-localhost:9435/metrics | fgrep cfs
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="1e-06"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="2e-06"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="4e-06"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="8e-06"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="1.6e-05"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="3.2e-05"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="6.4e-05"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.000128"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.000256"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.000512"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.001024"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.002048"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.004096"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.008192"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.016384"} 0
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.032768"} 18
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.065536"} 168
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.131072"} 168
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.262144"} 168
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="0.524288"} 168
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="1.048576"} 168
ebpf_exporter_cfs_throttling_seconds_bucket{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service",le="+Inf"} 168
ebpf_exporter_cfs_throttling_seconds_sum{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service"} 14.407136
ebpf_exporter_cfs_throttling_seconds_count{cgroup="/sys/fs/cgroup/system.slice/stress-ng.service"} 168
ebpf_exporter_ebpf_program_info{config="cfs-throttling",id="1312",program="unthrottle_cfs_rq",tag="3887b84da035bf7a"} 1
ebpf_exporter_enabled_configs{name="cfs-throttling"} 1
```
bobrik
commented
Oct 24, 2023
| SEC("fentry/unthrottle_cfs_rq") | ||
| int BPF_PROG(unthrottle_cfs_rq, struct cfs_rq *cfs_rq) | ||
| { | ||
| u64 throttled_us = (cfs_rq->rq->clock - cfs_rq->throttled_clock) / 1000; |
There was a problem hiding this comment.
cfs_rq->throttled_clock can be zero, at least in newer kernels:
|
Thanks to some tracing (#297, not yet committed demo), it is clear that things were not as they seemed: 
Even with 
Some more work is needed to properly understand this first. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Example throttled program:
One can observe throttling with
bpftrace:The values here also match what
cpu.statis reporting:Example output from
ebpf_exporter:cc @pims