Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v23.3.x] [CORE-5539] Pandaproxy: Avoid large allocations whilst serializing JSON #21469

Merged
merged 19 commits into from
Jul 22, 2024
Merged

[v23.3.x] [CORE-5539] Pandaproxy: Avoid large allocations whilst serializing JSON #21469

merged 19 commits into from
Jul 22, 2024

Conversation

BenPope
Copy link
Member

@BenPope BenPope commented Jul 17, 2024 •
edited
Loading

Backport of PR #20827

Fixes #21296

I took some refactoring changes from #17403 to make life simpler.

dotnwat and others added 19 commits July 17, 2024 11:34
@dotnwat @BenPope
json: factor out json specialization into submodule
6001bfe 
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
(cherry picked from commit dbf74db)

Conflicts:
  src/v/utils/json.h (was previously moved to container/json.h)
@dotnwat @BenPope
json: remove unused serializer
7db5143 
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
(cherry picked from commit 432cfb0)
@dotnwat @BenPope
json: move impl out of header
9aa686a 
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
(cherry picked from commit 20dcf97)
@dotnwat @BenPope
json: remove default template parameter
f4e2afa 
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
(cherry picked from commit 60f6717)
@BenPope
json: Template rjson_serialize on Buffer
4d4a296 
Some structures serialize quite large, so in order to avoid
oversize allocs, it's necessary to use a buffer type that is not
based onn contiguous memmory.

Explicitly instantiate all overloads with StringBuffer.

Fix the test due to the way name lookup works.

Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 7cea48d)
@BenPope
pandaproxy/json: Template rjson_serialize on Buffer
1afed6b 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit ebc4c13)
@BenPope
schema_registry: Template rjson_serialize on Buffer
463635b 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 9fcb311)
@BenPope
pandaproxy/json: Cleanup header includes
fa8392e 
Namespace changed due to lookup rules for templates.

Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit a455d23)

Conflicts: (all in header includes due to code motion)
  src/v/pandaproxy/json/requests/create_consumer.h
  src/v/pandaproxy/json/requests/fetch.h
  src/v/pandaproxy/json/requests/offset_fetch.h
  src/v/pandaproxy/json/requests/produce.h
@BenPope
pandaproxy/json: Avoid temporary for calls to write_body
01dca7c 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 5d90218)
@BenPope
schema_registry: Avoid temporary for calls to write_body
0a762b4 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit d74ec2a)

Conflicts:
  src/v/pandaproxy/schema_registry/handlers.cc (No JSON schema support)
@BenPope
json: Introduce chunked_buffer
24394ff 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit f17567f)
@BenPope
pandaproxy: Use chunked_buffer
51b76b0 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 744b0f0)

Conflicts:
  src/v/pandaproxy/json/requests/produce.h (header includes)
@BenPope
Revert "sr: Streaming json lists"
4958348 
This reverts commit 313d208.

Some edits required to avoid temporaries..

(cherry picked from commit 5f3236c)

Conflicts:
  src/v/pandaproxy/schema_registry/handlers.cc (header includes)
@BenPope
json: Introduce chunked_input_stream
1e0b1ee 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit c676f6e)
@BenPope
pandaproxy/json: Introduce chunked parsing with rjson_parse
929f422 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 240c682)
@BenPope
pandaproxy/rest: Use alternative rjson_parse
85275ef 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 8560e0a)
@BenPope
schema_registry: Use alternative rjson_parse
e708af6 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit 6795578)
@BenPope
pandaproxy: Enable streaming requests
fc8b1af 
Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit c934757)
@BenPope
pandaproxy/json: Move string version of rjson_parse
c6af45c 
It shouldn't be used in the general case.

Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit ac7c7dd)
@BenPope BenPope added area/schema-registry Schema Registry service within Redpanda kind/backport PRs targeting a stable branch labels Jul 17, 2024
@BenPope BenPope self-assigned this Jul 17, 2024
@BenPope BenPope added this to the v23.3.19 milestone Jul 17, 2024
@BenPope BenPope requested review from a team, aanthony-rp and oleiman and removed request for a team July 17, 2024 11:19
@BenPope
Copy link
Member Author

BenPope commented Jul 17, 2024 •
edited
Loading

Build failures both looks like:

Anyway, it's unrelated.

@BenPope BenPope mentioned this pull request Jul 17, 2024
Merged
7 tasks
aanthony-rp
aanthony-rp approved these changes Jul 18, 2024
View reviewed changes
Copy link
Contributor

@aanthony-rp aanthony-rp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assuming this is already reviewed code and just needs a bump. LMK if we need a deeper look.

@BenPope
Copy link
Member Author

BenPope commented Jul 18, 2024

assuming this is already reviewed code and just needs a bump. LMK if we need a deeper look.

Most of the conflicts were due to one of either:

  • Header movement, e.g., "seastarx.h" moved to "base/seastarx.h" - and they got shuffled around by sort order.
  • Changes to JSON Schema support, which is not in the branch

@piyushredpanda
Copy link
Contributor

Force-merging given known failures.

@piyushredpanda piyushredpanda merged commit deeac3a into redpanda-data:v23.3.x Jul 22, 2024
14 of 18 checks passed
@BenPope BenPope mentioned this pull request Jul 30, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aanthony-rp aanthony-rp aanthony-rp approved these changes

@oleiman oleiman oleiman approved these changes

Assignees

@BenPope BenPope

Labels
area/redpanda area/schema-registry Schema Registry service within Redpanda kind/backport PRs targeting a stable branch
Projects
None yet
Milestone
v23.3.19
Development

Successfully merging this pull request may close these issues.
5 participants
@BenPope @piyushredpanda @oleiman @aanthony-rp @dotnwat