gh-129005: Move bytearray to use bytes as a buffer #130563
Closed
+65
−40
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
tp_new
to guaranteeob_bytes_head
is always set, often to the empty bytes singleton.ob_alloc
information is now redundant, added assertion to validate that, would it make sense to deprecate?bytearray
code very similar tobytes
code, more could likely be just proxied to thebytes
now (Ex. construction from an object in__new__
). Here just focusing on the swap as that enables optimizations.bytearray
is passed a single-reference bytes it could potentially take "ownership" of it without copying the bytes, for now not implemented.This enables adding
bytearray._detach()
which I plan to do in a separate PR.TODO
_io.FileIO.readall()
can't run into that case as it never passes in a string to start with, just a size.Performance
On my machine (AMD 64 bit Linux, Optimized build):
_io
takes: ~0.791s and uses ~2GB of RAM_pyio
current: ~1.073s and uses ~4GB of RAM_pyio
withbytearray._detach
: ~0.887s and uses ~2GB of RAMPerf checking no major swings in an optimized build:
./python -E -bb -Wd -m test -uall -M32G test_bytes test_capi.test_bytearray -vvv
before: ~1.4s
after: ~1.4s
Previous discussion: https://discuss.python.org/t/add-zero-copy-conversion-of-bytearray-to-bytes-by-providing-bytes/79164
cc: @serhiy-storchaka @vstinner