Raise loop and chunk limit for newer kernels #1795
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benkilimnik
force-pushed
the
kernel_chunk
branch
from
547ba54
to
e4697eb
Compare
Signed-off-by: Benjamin Kilimnik <bkilimnik@pixielabs.ai>
benkilimnik
force-pushed
the
kernel_chunk
branch
from
e4697eb
to
47b7bef
Compare
benkilimnik
changed the title
benkilimnik
changed the title
Signed-off-by: Benjamin Kilimnik <bkilimnik@pixielabs.ai>
benkilimnik
force-pushed
the
kernel_chunk
branch
from
09409f1
to
dd11bcb
Compare
Signed-off-by: Benjamin Kilimnik <bkilimnik@pixielabs.ai>
| @@ -431,6 +432,20 @@ auto SocketTraceConnector::InitPerfBufferSpecs() { | |||
| } | |||
|
|
|||
| Status SocketTraceConnector::InitBPF() { | |||
| // set BPF loop limit and chunk limit based on kernel version | |||
| auto kernel = system::GetCachedKernelVersion(); | |||
| int loop_limit = 42; | |||
Signed-off-by: Benjamin Kilimnik <bkilimnik@pixielabs.ai>
benkilimnik
force-pushed
the
kernel_chunk
branch
from
bb6822d
to
2a7f0f5
Compare
etep
approved these changes
Jan 18, 2024
JamesMBartlett
approved these changes
Jan 23, 2024
| // Kernels >= 5.1 have higher BPF instruction limits (1 million for verifier). | ||
| // This enables a 21x increase to our loop and chunk limits | ||
| FLAGS_stirling_bpf_loop_limit = 882; | ||
| FLAGS_stirling_bpf_chunk_limit = 84; |
There was a problem hiding this comment.
How did you pick these numbers? How close are we to the instruction limit with these numbers?
There was a problem hiding this comment.
I manually raised the loop/chunk limits to see when the verifier would throw an error and chose values just below that upper bound, which is around 22x our previous limits.
bpf: Argument list too long. Program too large (... insns), at most 4096 insnsConfusingly, the bpf(2) syscall returns the same error whether the program size or its complexity exceeds the limits. See stackoverflow
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Dynamically increase the loop limit for newer kernels with higher instruction limits (1 million for kernels > 5.1) by 21x to reduce data loss and raise ingest. More details in #1755.
One open question is whether we want to add vizier flag to toggle this behavior in case there are unforseen performance bottlenecks for certain clusters.
Type of change: /kind feature
Test Plan: Existing targets + perf/demo tests outlined in #1755.