-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PQ: Event containing 20+MB String cannot be deserialized for processing #16683
Comments
I can confirm that the static initialization of the The existing ruby implementation helpfully logs as it goes (and these logs are covered in specs), but since the |
I suspect that our inability to reproduce has to do with the recent reversion of changes to the buffered tokenizer, which had allowed over-large payloads (after reverting, I believe they are truncated), since my repro used stdin, which "conveniently" auto-upgrades from
I'm attempting to reproduce without routing through that. |
When Logstash 8.12.0 added increased Jackson stream read constraints to jvm.options, assumptions about the existence of that file's contents were invalidated. This led to issues like elastic#16683. This change ensures Logstash applies defaults from config at runtime: - MAX_STRING_LENGTH: 200_000_000 - MAX_NUMBER_LENGTH: 10_000 - MAX_NESTING_DEPTH: 1_000 These match the jvm.options defaults and are applied even when config is missing. Config values still override these defaults when present.
When Logstash 8.12.0 added increased Jackson stream read constraints to jvm.options, assumptions about the existence of that file's contents were invalidated. This led to issues like elastic#16683. This change ensures Logstash applies defaults from config at runtime: - MAX_STRING_LENGTH: 200_000_000 - MAX_NUMBER_LENGTH: 10_000 - MAX_NESTING_DEPTH: 1_000 These match the jvm.options defaults and are applied even when config is missing. Config values still override these defaults when present.
When Logstash 8.12.0 added increased Jackson stream read constraints to jvm.options, assumptions about the existence of that file's contents were invalidated. This led to issues like #16683. This change ensures Logstash applies defaults from config at runtime: - MAX_STRING_LENGTH: 200_000_000 - MAX_NUMBER_LENGTH: 10_000 - MAX_NESTING_DEPTH: 1_000 These match the jvm.options defaults and are applied even when config is missing. Config values still override these defaults when present.
When Logstash 8.12.0 added increased Jackson stream read constraints to jvm.options, assumptions about the existence of that file's contents were invalidated. This led to issues like #16683. This change ensures Logstash applies defaults from config at runtime: - MAX_STRING_LENGTH: 200_000_000 - MAX_NUMBER_LENGTH: 10_000 - MAX_NESTING_DEPTH: 1_000 These match the jvm.options defaults and are applied even when config is missing. Config values still override these defaults when present. (cherry picked from commit cc608eb)
When Logstash 8.12.0 added increased Jackson stream read constraints to jvm.options, assumptions about the existence of that file's contents were invalidated. This led to issues like #16683. This change ensures Logstash applies defaults from config at runtime: - MAX_STRING_LENGTH: 200_000_000 - MAX_NUMBER_LENGTH: 10_000 - MAX_NESTING_DEPTH: 1_000 These match the jvm.options defaults and are applied even when config is missing. Config values still override these defaults when present. (cherry picked from commit cc608eb)
When Logstash 8.12.0 added increased Jackson stream read constraints to jvm.options, assumptions about the existence of that file's contents were invalidated. This led to issues like #16683. This change ensures Logstash applies defaults from config at runtime: - MAX_STRING_LENGTH: 200_000_000 - MAX_NUMBER_LENGTH: 10_000 - MAX_NESTING_DEPTH: 1_000 These match the jvm.options defaults and are applied even when config is missing. Config values still override these defaults when present. (cherry picked from commit cc608eb)
…16846) When Logstash 8.12.0 added increased Jackson stream read constraints to jvm.options, assumptions about the existence of that file's contents were invalidated. This led to issues like #16683. This change ensures Logstash applies defaults from config at runtime: - MAX_STRING_LENGTH: 200_000_000 - MAX_NUMBER_LENGTH: 10_000 - MAX_NESTING_DEPTH: 1_000 These match the jvm.options defaults and are applied even when config is missing. Config values still override these defaults when present. (cherry picked from commit cc608eb) Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
…16845) When Logstash 8.12.0 added increased Jackson stream read constraints to jvm.options, assumptions about the existence of that file's contents were invalidated. This led to issues like #16683. This change ensures Logstash applies defaults from config at runtime: - MAX_STRING_LENGTH: 200_000_000 - MAX_NUMBER_LENGTH: 10_000 - MAX_NESTING_DEPTH: 1_000 These match the jvm.options defaults and are applied even when config is missing. Config values still override these defaults when present. (cherry picked from commit cc608eb) Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
…16847) When Logstash 8.12.0 added increased Jackson stream read constraints to jvm.options, assumptions about the existence of that file's contents were invalidated. This led to issues like #16683. This change ensures Logstash applies defaults from config at runtime: - MAX_STRING_LENGTH: 200_000_000 - MAX_NUMBER_LENGTH: 10_000 - MAX_NESTING_DEPTH: 1_000 These match the jvm.options defaults and are applied even when config is missing. Config values still override these defaults when present. (cherry picked from commit cc608eb) Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
Logstash information:
Please include the following information:
bin/logstash --version
)8.x
,8.15
,main
Plugins installed: (
bin/logstash-plugin list --verbose
): N/AJVM (e.g.
java -version
): AnyOS version (
uname -a
if on a Unix-like system): AnyDescription of the problem including expected versus actual behavior:
Steps to reproduce:
Provide logs (if relevant):
(using improved pipeline cause-chain logging from #16677):
Notably:
Notably, setting -Dlogstash.jackson.stream-read-constraints.max-string-length=400000000 does NOT change the error message, indicating that the stream constraint is not applied to the specific deserializer that works with the PQ, possibly because the
CBOR_MAPPER
is configred before the overrides are applied.The text was updated successfully, but these errors were encountered: