fix(c/driver/postgresql): chunk large COPY payloads #1937
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PostgreSQL apparently has an internal limit - split up batches to stay under that limit. It doesn't care about message boundaries in this mode, so we can chunk naively. Fixes apache#1921.
paleolimbot
approved these changes
Jun 25, 2024
c/driver/postgresql/statement.cc
Outdated
Comment on lines
575
to
578
| // https://github.com/apache/arrow-adbc/issues/1921: PostgreSQL has a max | ||
| // size for a single message that we need to respect (1 GiB - 1). Since | ||
| // the buffer can be chunked up as much as we want, go for 512 MiB as our | ||
| // limit. |
There was a problem hiding this comment.
Is there any advantage to having it be this big from our end? Maybe it's buffered internally in libpq and it doesn't matter, but my intuition might have been something more like 16 MB here (surely it's limited by the network and not our loop).
There was a problem hiding this comment.
It's pretty arbitrary. I shrunk it to 16MiB.
zeroshade
approved these changes
Jun 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PostgreSQL apparently has an internal limit - split up batches to stay under that limit. It doesn't care about message boundaries in this mode, so we can chunk naively.
Fixes #1921.