Improve cpu/memory (proctree wise) #4503

geyslan · 2025-01-14T11:21:51Z

1. Explain what the PR does

Running local proctree stressor (deterministic workload) we got these results:

| **Metric**         | **main**  | **new**   | **Difference (%)** |
| ------------------ | --------- | --------- | ------------------ |
| **CPU Peak (%)**   | 3.51%     | 3.27%     | -6.84%             |
| **CPU Avg (%)**    | 2.93%     | 2.72%     | -7.17%             |
| **Malloc Rate**    | 2,143,174 | 1,884,787 | -12.06%            |
| **Heap Peak (MB)** | 747 MB    | 694 MB    | -7.09%             |
| **Heap Avg (MB)**  | 258 MB    | 229 MB    | -11.24%            |
| **Heap Obj Avg**   | 1,102,839 | 1,101,027 | -0.16%             |

Tracee flags:

-e sched_process_exec,sched_process_fork,sched_process_exit --proctree source=both --proctree process-cache=16384 --proctree thread-cache=32768 --proctree disable-procfs -o none

Stressor details:

8 threads with 2_000_000 ops each

c1dd27e chore(cmd): add proctree disable-procfs
aa98b6c perf(controlplane): introduce signal pool
6c255f3 perf(proctree): improve Process concurrency ctrl
a90089f perf(proctree): change Thread concurrency control
dbb0bd0 perf(proctree): reduce lock contention
3c9b20e chore(proctree): remove leftover
6fd5d10 chore/perf(proctree): comment out exit fields
6bd9461 perf(proctree): introduce feed pools
e120838 perf(proctree): move functions from FeedFromFork
e10c562 perf(events): improve ArgVal
b2638f6 chore(events): add BenchmarkArgVal
c585dcb perf: improve procTreeExitProcessor
56e011d chore: add Benchmark_procTreeExitProcessor
adad405 perf: remove unused ExecFeed interpreter fields
6eba0fd perf(controlplane): improve procTreeExecProcessor
32464dc chore(controlplane): add Benchmark_procTreeExecProcessor
54ec91d perf(ebpf): improve procTreeExecProcessor
7f7a34c chore(ebpf): add Benchmark_procTreeExecProcessor
6dfc496 perf(controlplane): improve procTreeForkProcessor
f3f7fa4 chore(controlplane): add procTreeForkProcessor bench
e7c490c perf(ebpf): improve procTreeForkProcessor
b28a532 chore(ebpf): add Benchmark_procTreeForkProcessor
db87a1a perf: reduce events.Core lock contention
e3e2e67 chore(bufferdecoder): set zero from def fields
baced22 chore(bufferdecode): add DecodeArguments benchmark

aa98b6c perf(controlplane): introduce signal pool

It helps to reduce the stack dynamic growth and the number of
allocations, which is good for performance.

a90089f perf(proctree): change Thread concurrency control

Mutex is a heavy lock, and it's not necessary to use it in the Thread
concurrency control. This change replaces the mutex with atomic
operations to reduce contention, what also reduces memory footprint.

dbb0bd0 perf(proctree): reduce lock contention

Reuse the same TaskInfo reference avoiding the need to lock to fetch it.

This also reorders the creation of the process and thread.

6fd5d10 chore/perf(proctree): comment out exit fields

The unique ExitFeed fields being tackeld by FeedFromExit() are
TaskHash and TimeStamp. Then this commit comments out the other fields
that are not being used by the proctree in this context.

6bd9461 perf(proctree): introduce feed pools

It helps to reduce the stack dynamic growth and the number of
allocations, which is good for performance.

Changelog fields now holds pointers to the feeds, instead of the feeds
themselves. This way, it aligns with the new feed pointers avoiding
de-referencing.

e10c562 perf(events): improve ArgVal

| Sub-Benchmark    | Old (ns/op) | New (ns/op) | Change (%) |
|------------------|-------------|-------------|------------|
| valid_args       | 14.43       | 13.35       | -7.48%     |
| invalid_val_type | 551.7       | 589.8       | +6.90%     |
| not_found_arg    | 499.2       | 586.0       | +17.38%    |

The valid_args is the most relevant case, since it traverses args based
on a specific order. The other cases are not deterministic and used to
measure upcoming changes for the worst case.

---

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^BenchmarkArgVal$
github.com/aquasecurity/tracee/pkg/events/parse -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/events/parse
cpu: AMD Ryzen 9 7950X 16-Core Processor
BenchmarkArgVal/int32/valid_args-32        100000000   13.35 ns/op   0 B/op   0 allocs/op
BenchmarkArgVal/int32/invalid_val_type-32  100000000  589.8 ns/op  584 B/op  10 allocs/op
BenchmarkArgVal/int32/not_found_arg-32     100000000  586.0 ns/op  520 B/op  10 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/events/parse  118.922s

b2638f6 chore(events): add BenchmarkArgVal

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^BenchmarkArgVal$
github.com/aquasecurity/tracee/pkg/events/parse -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/events/parse
cpu: AMD Ryzen 9 7950X 16-Core Processor
BenchmarkArgVal/int32/valid_args-32        100000000   14.43 ns/op   0 B/op  0 allocs/op
BenchmarkArgVal/int32/invalid_val_type-32  100000000  551.7 ns/op  584 B/op 10 allocs/op
BenchmarkArgVal/int32/not_found_arg-32     100000000  499.2 ns/op  520 B/op 10 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/events/parse  106.538s

c585dcb perf: improve procTreeExitProcessor

Improve procTreeExitProcessor for both Tracee and Controller.

-

Tracee

| Metric                  | Old Value  | New Value   | Improvement (%) |
|-------------------------|------------|-------------|-----------------|
| Time per operation (ns) | 159.9      | 95.71       | 40.14%          |
| Bytes allocated (B/op)  | 48         | 0           | 100.00%         |
| Allocations per op      | 2          | 0           | 100.00%         |
| Total runtime (s)       | 16.001     | 9.586       | 40.14%          |

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeExitProcessor$
github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeExitProcessor-32  100000000  95.71 ns/op  0 B/op  0 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf  9.586s

---

Controller

| Metric                  | Old Value  | New Value   | Improvement (%) |
|-------------------------|------------|-------------|-----------------|
| Time per operation (ns) | 335.5      | 115.4       | 65.60%          |
| Bytes allocated (B/op)  | 240        | 0           | 100.00%         |
| Allocations per op      | 4          | 0           | 100.00%         |
| Total runtime (s)       | 33.558     | 11.553      | 65.60%          |

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeExitProcessor$
github.com/aquasecurity/tracee/pkg/ebpf/controlplane -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeExitProcessor-32  100000000  115.4 ns/op  0 B/op  0 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf/controlplane  11.553s

56e011d chore: add Benchmark_procTreeExitProcessor

For both Tracee and Controller.

-

Tracee

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeExitProcessor$
github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeExitProcessor-32  100000000  159.9 ns/op  48 B/op  2 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf  16.001s

---

Controller

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeExitProcessor$
github.com/aquasecurity/tracee/pkg/ebpf/controlplane -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeExitProcessor-32  100000000 335.5 ns/op  240 B/op  4 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf/controlplane  33.558s

adad405 perf: remove unused ExecFeed interpreter fields

Disable (comment out) ExecFeed interpreter fields not used by the
feeders. This removal was already started by 4a5bb5d0f.

---

Tracee

| Metric                  | Old Value  | New Value   | Improvement (%) |
|-------------------------|------------|-------------|-----------------|
| Time per operation (ns) | 215.6      | 168.1       | 22.03%          |
| Bytes allocated (B/op)  | 4          | 4           | 0.00%           |
| Allocations per op      | 1          | 1           | 0.00%           |
| Total runtime (s)       | 21.571     | 16.825      | 22.03%          |

-

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$
github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeExecProcessor-32  100000000  168.1 ns/op  4 B/op 1 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf  16.825s

---

Controller

| Metric                  | Old Value  | New Value   | Improvement (%) |
|-------------------------|------------|-------------|-----------------|
| Time per operation (ns) | 284.2      | 209.7       | 26.20%          |
| Bytes allocated (B/op)  | 4          | 4           | 0.00%           |
| Allocations per op      | 1          | 1           | 0.00%           |
| Total runtime (s)       | 28.435     | 20.983      | 26.20%          |

-

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$
github.com/aquasecurity/tracee/pkg/ebpf/controlplane -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeExecProcessor-32  100000000  209.7 ns/op  4 B/op  1 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf/controlplane  20.983s

6eba0fd perf(controlplane): improve procTreeExecProcessor

| Metric                  | Old Value  | New Value   | Improvement (%) |
|-------------------------|------------|-------------|-----------------|
| Time per operation (ns) | 649.7      | 284.2       | 56.26%          |
| Bytes allocated (B/op)  | 500        | 4           | 99.20%          |
| Allocations per op      | 6          | 1           | 83.33%          |
| Total runtime (s)       | 64.981     | 28.435      | 56.26%          |

---

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$
github.com/aquasecurity/tracee/pkg/ebpf/controlplane -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeExecProcessor-32  100000000  284.2 ns/op  4 B/op  1 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf/controlplane  28.435s

32464dc chore(controlplane): add Benchmark_procTreeExecProcessor

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$
github.com/aquasecurity/tracee/pkg/ebpf/controlplane
-benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeExecProcessor-32 100000000  649.7 ns/op  500 B/op  6 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf/controlplane  64.981s

54ec91d perf(ebpf): improve procTreeExecProcessor

| Metric                  | Old Value  | New Value   | Improvement (%) |
|-------------------------|------------|-------------|-----------------|
| Time per operation (ns) | 514.7      | 215.6       | 58.12%          |
| Bytes allocated (B/op)  | 500        | 4           | 99.20%          |
| Allocations per op      | 6          | 1           | 83.33%          |
| Total runtime (s)       | 51.483     | 21.571      | 58.12%          |

---

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$
github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeExecProcessor-32  100000000  215.6 ns/op  4 B/op  1 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf  21.571s

7f7a34c chore(ebpf): add Benchmark_procTreeExecProcessor

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$
github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeExecProcessor-32 100000000  514.7 ns/op  500 B/op  6 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf  51.483s

6dfc496 perf(controlplane): improve procTreeForkProcessor

| Metric                  | Old Value  | New Value   | Improvement (%) |
|-------------------------|------------|-------------|-----------------|
| Time per operation (ns) | 618.2      | 274.0       | 55.67%          |
| Bytes allocated (B/op)  | 496        | 0           | 100.00%         |
| Allocations per op      | 5          | 0           | 100.00%         |
| Total runtime (s)       | 61.827     | 27.415      | 55.67%          |

---

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeForkProcessor$
github.com/aquasecurity/tracee/pkg/ebpf/controlplane
-benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeForkProcessor-32  100000000  274.0 ns/op  0 B/op  0 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf/controlplane  27.415s

f3f7fa4 chore(controlplane): add procTreeForkProcessor bench

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeForkProcessor$
github.com/aquasecurity/tracee/pkg/ebpf/controlplane
-benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeForkProcessor-32 100000000  618.2 ns/op  496 B/op  5 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf/controlplane  61.827s

e7c490c perf(ebpf): improve procTreeForkProcessor

| Metric                  | Old Value  | New Value   | Improvement (%) |
|-------------------------|------------|-------------|-----------------|
| Time per operation (ns) | 547.4      | 267.5       | 51.14%          |
| Bytes allocated (B/op)  | 496        | 0           | 100.00%         |
| Allocations per op      | 5          | 0           | 100.00%         |
| Total runtime (s)       | 54.757     | 26.763      | 51.13%          |

---

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeForkProcessor$
github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeForkProcessor-32 100000000  267.5 ns/op  0 B/op  0 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf  26.763s

b28a532 chore(ebpf): add Benchmark_procTreeForkProcessor

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^Benchmark_procTreeForkProcessor$
github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/ebpf
cpu: AMD Ryzen 9 7950X 16-Core Processor
Benchmark_procTreeForkProcessor-32 100000000  547.4 ns/op  496 B/op  5 allocs/op
PASS
ok  github.com/aquasecurity/tracee/pkg/ebpf  54.757s

db87a1a perf: reduce events.Core lock contention

When retrieving the event definition, there is no longer a need to check
beforehand Core.IsDefined(). Validation can now be performed directly
using the NotValid() method on the Definition type returned by
GetEventDefinitionID() and GetEventDefinitionName().

Besides the lock contention reduction, this also gets rid of the window
where the event definition could be changed between the check and the
actual use of the definition.

This also fixes a wrong logger usage in the pipeline.

e3e2e67 chore(bufferdecoder): set zero from def fields

It's a cosmetic change to make the code more readable.

baced22 chore(bufferdecode): add DecodeArguments benchmark

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem
-run=^$ -tags ebpf -bench ^BenchmarkDecodeArguments$
github.com/aquasecurity/tracee/pkg/bufferdecoder -benchtime=100000000x

goos: linux
goarch: amd64
pkg: github.com/aquasecurity/tracee/pkg/bufferdecoder
cpu: AMD Ryzen 9 7950X 16-Core Processor
BenchmarkDecodeArguments-32 100000000  206.3 ns/op  512 B/op  1 alloc/op
PASS
ok  github.com/aquasecurity/tracee/pkg/bufferdecoder  20.646s

2. Explain how to test it

3. Other comments

pkg/ebpf/event_parameters.go

pkg/ebpf/events_pipeline.go

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^BenchmarkDecodeArguments$ github.com/aquasecurity/tracee/pkg/bufferdecoder -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/bufferdecoder cpu: AMD Ryzen 9 7950X 16-Core Processor BenchmarkDecodeArguments-32 100000000 206.3 ns/op 512 B/op 1 alloc/op PASS ok github.com/aquasecurity/tracee/pkg/bufferdecoder 20.646s

It's a cosmetic change to make the code more readable.

When retrieving the event definition, there is no longer a need to check beforehand Core.IsDefined(). Validation can now be performed directly using the NotValid() method on the Definition type returned by GetEventDefinitionID() and GetEventDefinitionName(). Besides the lock contention reduction, this also gets rid of the window where the event definition could be changed between the check and the actual use of the definition. This also fixes a wrong logger usage in the pipeline.

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeForkProcessor$ github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeForkProcessor-32 100000000 547.4 ns/op 496 B/op 5 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf 54.757s

| Metric | Old Value | New Value | Improvement (%) | |-------------------------|------------|-------------|-----------------| | Time per operation (ns) | 547.4 | 267.5 | 51.14% | | Bytes allocated (B/op) | 496 | 0 | 100.00% | | Allocations per op | 5 | 0 | 100.00% | | Total runtime (s) | 54.757 | 26.763 | 51.13% | --- Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeForkProcessor$ github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeForkProcessor-32 100000000 267.5 ns/op 0 B/op 0 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf 26.763s

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeForkProcessor$ github.com/aquasecurity/tracee/pkg/ebpf/controlplane -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeForkProcessor-32 100000000 618.2 ns/op 496 B/op 5 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf/controlplane 61.827s

| Metric | Old Value | New Value | Improvement (%) | |-------------------------|------------|-------------|-----------------| | Time per operation (ns) | 618.2 | 274.0 | 55.67% | | Bytes allocated (B/op) | 496 | 0 | 100.00% | | Allocations per op | 5 | 0 | 100.00% | | Total runtime (s) | 61.827 | 27.415 | 55.67% | --- Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeForkProcessor$ github.com/aquasecurity/tracee/pkg/ebpf/controlplane -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeForkProcessor-32 100000000 274.0 ns/op 0 B/op 0 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf/controlplane 27.415s

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$ github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeExecProcessor-32 100000000 514.7 ns/op 500 B/op 6 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf 51.483s

| Metric | Old Value | New Value | Improvement (%) | |-------------------------|------------|-------------|-----------------| | Time per operation (ns) | 514.7 | 215.6 | 58.12% | | Bytes allocated (B/op) | 500 | 4 | 99.20% | | Allocations per op | 6 | 1 | 83.33% | | Total runtime (s) | 51.483 | 21.571 | 58.12% | --- Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$ github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeExecProcessor-32 100000000 215.6 ns/op 4 B/op 1 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf 21.571s

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$ github.com/aquasecurity/tracee/pkg/ebpf/controlplane -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeExecProcessor-32 100000000 649.7 ns/op 500 B/op 6 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf/controlplane 64.981s

| Metric | Old Value | New Value | Improvement (%) | |-------------------------|------------|-------------|-----------------| | Time per operation (ns) | 649.7 | 284.2 | 56.26% | | Bytes allocated (B/op) | 500 | 4 | 99.20% | | Allocations per op | 6 | 1 | 83.33% | | Total runtime (s) | 64.981 | 28.435 | 56.26% | --- Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$ github.com/aquasecurity/tracee/pkg/ebpf/controlplane -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeExecProcessor-32 100000000 284.2 ns/op 4 B/op 1 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf/controlplane 28.435s

Disable (comment out) ExecFeed interpreter fields not used by the feeders. This removal was already started by 4a5bb5d. --- Tracee | Metric | Old Value | New Value | Improvement (%) | |-------------------------|------------|-------------|-----------------| | Time per operation (ns) | 215.6 | 168.1 | 22.03% | | Bytes allocated (B/op) | 4 | 4 | 0.00% | | Allocations per op | 1 | 1 | 0.00% | | Total runtime (s) | 21.571 | 16.825 | 22.03% | - Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$ github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeExecProcessor-32 100000000 168.1 ns/op 4 B/op 1 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf 16.825s --- Controller | Metric | Old Value | New Value | Improvement (%) | |-------------------------|------------|-------------|-----------------| | Time per operation (ns) | 284.2 | 209.7 | 26.20% | | Bytes allocated (B/op) | 4 | 4 | 0.00% | | Allocations per op | 1 | 1 | 0.00% | | Total runtime (s) | 28.435 | 20.983 | 26.20% | - Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeExecProcessor$ github.com/aquasecurity/tracee/pkg/ebpf/controlplane -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeExecProcessor-32 100000000 209.7 ns/op 4 B/op 1 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf/controlplane 20.983s

For both Tracee and Controller. - Tracee Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeExitProcessor$ github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeExitProcessor-32 100000000 159.9 ns/op 48 B/op 2 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf 16.001s --- Controller Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeExitProcessor$ github.com/aquasecurity/tracee/pkg/ebpf/controlplane -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeExitProcessor-32 100000000 335.5 ns/op 240 B/op 4 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf/controlplane 33.558s

Improve procTreeExitProcessor for both Tracee and Controller. - Tracee | Metric | Old Value | New Value | Improvement (%) | |-------------------------|------------|-------------|-----------------| | Time per operation (ns) | 159.9 | 95.71 | 40.14% | | Bytes allocated (B/op) | 48 | 0 | 100.00% | | Allocations per op | 2 | 0 | 100.00% | | Total runtime (s) | 16.001 | 9.586 | 40.14% | Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeExitProcessor$ github.com/aquasecurity/tracee/pkg/ebpf -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeExitProcessor-32 100000000 95.71 ns/op 0 B/op 0 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf 9.586s --- Controller | Metric | Old Value | New Value | Improvement (%) | |-------------------------|------------|-------------|-----------------| | Time per operation (ns) | 335.5 | 115.4 | 65.60% | | Bytes allocated (B/op) | 240 | 0 | 100.00% | | Allocations per op | 4 | 0 | 100.00% | | Total runtime (s) | 33.558 | 11.553 | 65.60% | Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_procTreeExitProcessor$ github.com/aquasecurity/tracee/pkg/ebpf/controlplane -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/ebpf/controlplane cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_procTreeExitProcessor-32 100000000 115.4 ns/op 0 B/op 0 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/ebpf/controlplane 11.553s

Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^BenchmarkArgVal$ github.com/aquasecurity/tracee/pkg/events/parse -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/events/parse cpu: AMD Ryzen 9 7950X 16-Core Processor BenchmarkArgVal/int32/valid_args-32 100000000 14.43 ns/op 0 B/op 0 allocs/op BenchmarkArgVal/int32/invalid_val_type-32 100000000 551.7 ns/op 584 B/op 10 allocs/op BenchmarkArgVal/int32/not_found_arg-32 100000000 499.2 ns/op 520 B/op 10 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/events/parse 106.538s

| Sub-Benchmark | Old (ns/op) | New (ns/op) | Change (%) | |------------------|-------------|-------------|------------| | valid_args | 14.43 | 13.35 | -7.48% | | invalid_val_type | 551.7 | 589.8 | +6.90% | | not_found_arg | 499.2 | 586.0 | +17.38% | The valid_args is the most relevant case, since it traverses args based on a specific order. The other cases are not deterministic and used to measure upcoming changes for the worst case. --- Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^BenchmarkArgVal$ github.com/aquasecurity/tracee/pkg/events/parse -benchtime=100000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/events/parse cpu: AMD Ryzen 9 7950X 16-Core Processor BenchmarkArgVal/int32/valid_args-32 100000000 13.35 ns/op 0 B/op 0 allocs/op BenchmarkArgVal/int32/invalid_val_type-32 100000000 589.8 ns/op 584 B/op 10 allocs/op BenchmarkArgVal/int32/not_found_arg-32 100000000 586.0 ns/op 520 B/op 10 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/events/parse 118.922s

It helps to reduce the stack dynamic growth and the number of allocations, which is good for performance. Changelog fields now holds pointers to the feeds, instead of the feeds themselves. This way, it aligns with the new feed pointers avoiding de-referencing.

The unique ExitFeed fields being tackeld by FeedFromExit() are TaskHash and TimeStamp. Then this commit comments out the other fields that are not being used by the proctree in this context.

Reuse the same TaskInfo reference avoiding the need to lock to fetch it. This also reorders the creation of the process and thread.

Mutex is a heavy lock, and it's not necessary to use it in the Thread concurrency control. This change replaces the mutex with atomic operations to reduce contention, what also reduces memory footprint.

It helps to reduce the stack dynamic growth and the number of allocations, which is good for performance.

geyslan added kind/bug area/performance labels Jan 14, 2025

geyslan self-assigned this Jan 14, 2025

github-actions bot added area/ebpf area/testing area/events area/grpc and removed area/performance labels Jan 14, 2025

geyslan changed the title ~~Improv cpu time~~ Improve cpu time Jan 14, 2025

github-advanced-security bot found potential problems Jan 14, 2025

View reviewed changes

pkg/ebpf/event_parameters.go Fixed Show resolved Hide resolved

geyslan force-pushed the improv-cpu-time branch 4 times, most recently from cb59e3e to cde5a80 Compare January 17, 2025 00:38

geyslan changed the title ~~Improve cpu time~~ Improve cpu/memory (proctree wise) Jan 17, 2025

geyslan added the milestone/v0.23.0 label Jan 17, 2025

geyslan force-pushed the improv-cpu-time branch from cde5a80 to e4a750b Compare January 17, 2025 16:41

geyslan requested review from NDStrahilevitz, rscampos and yanivagman January 17, 2025 17:30

geyslan marked this pull request as ready for review January 17, 2025 17:31

geyslan force-pushed the improv-cpu-time branch from e4a750b to 5e3ad59 Compare January 17, 2025 17:34

github-actions bot added kind/documentation area/UX area/flags labels Jan 18, 2025

geyslan commented Jan 19, 2025

View reviewed changes

pkg/ebpf/events_pipeline.go Outdated Show resolved Hide resolved

geyslan added 4 commits January 22, 2025 19:17

chore(bufferdecoder): set zero from def fields

e3e2e67

It's a cosmetic change to make the code more readable.

geyslan added 21 commits January 22, 2025 19:28

perf(proctree): move functions from FeedFromFork

e120838

chore/perf(proctree): comment out exit fields

6fd5d10

The unique ExitFeed fields being tackeld by FeedFromExit() are TaskHash and TimeStamp. Then this commit comments out the other fields that are not being used by the proctree in this context.

chore(proctree): remove leftover

3c9b20e

perf(proctree): reduce lock contention

dbb0bd0

Reuse the same TaskInfo reference avoiding the need to lock to fetch it. This also reorders the creation of the process and thread.

perf(proctree): change Thread concurrency control

a90089f

Mutex is a heavy lock, and it's not necessary to use it in the Thread concurrency control. This change replaces the mutex with atomic operations to reduce contention, what also reduces memory footprint.

perf(proctree): improve Process concurrency ctrl

6c255f3

perf(controlplane): introduce signal pool

aa98b6c

It helps to reduce the stack dynamic growth and the number of allocations, which is good for performance.

chore(cmd): add proctree disable-procfs

c1dd27e

geyslan force-pushed the improv-cpu-time branch from 314326e to c1dd27e Compare January 22, 2025 22:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve cpu/memory (proctree wise) #4503

Improve cpu/memory (proctree wise) #4503

geyslan commented Jan 14, 2025 •

edited

Loading

Improve cpu/memory (proctree wise) #4503

Are you sure you want to change the base?

Improve cpu/memory (proctree wise) #4503

Conversation

geyslan commented Jan 14, 2025 • edited Loading

1. Explain what the PR does

2. Explain how to test it

3. Other comments

geyslan commented Jan 14, 2025 •

edited

Loading