Support writing any bytes-like object to S3. #361
Conversation
|
All the tests pass, so I guess that check was not necessary. Good catch! Could you please update the docstring of that method and mention the Buffer interface? Is it possible to write a unit test for this without including PyArrow? |
|
I don't remember that " Just to make sure we're not regressing somewhere. |
gilbsgilbs
force-pushed
the
s3-support-any-bytelike-object
branch
from
97d4fc5 to
423d7db
Compare
It comes from this commit which I don't think is relevant: 5f3b6fa8#diff-6d8b12a1a0fab36149bf68fc0ef770faR315 It was then improved in #293 to support memoryviews.
Not sure how 🤔 . I think I'd have to implement the buffer interface in a C extension somehow which sounds overkill. With pyarrow, it'd be quite easy to reproduce, still you'd have to download loads of wheels or even compile it yourself which also sounds overkill for the tests. import pyarrow as pa
import smart_open
buf = pa.py_buffer(b'foobar')
with open('raw', 'wb') as f:
f.write(buf) # Works
with smart_open.open('s3://…', 'wb') as f:
f.write(buf) # Crashes
✔️ |
|
@piskvorky @mpenkov anything else I can do? |
| @@ -539,22 +533,19 @@ def detach(self): | |||
| raise io.UnsupportedOperation("detach() not supported") | |||
|
|
|||
| def write(self, b): | |||
| """Write the given bytes (binary string) to the S3 file. | |||
| """Write the given buffer (bytes, bytearray, memoryview or any buffer | |||
| interface implementation) to the S3 file. | |||
There was a problem hiding this comment.
Can you please include a link to the appopriate standard library documentation?
Is it this one? https://docs.python.org/3/c-api/buffer.html
There was a problem hiding this comment.
Yes, that's the right one.
|
Congrats on your first contribution to smart_open @gilbsgilbs 🥇 !!! |
|
Thanks for the prompt review and merge! May I ask when you expect to release it? This is quite blocking for me actually. |
|
We will try to release before the end of October. |
Fixes #360