support binary file handles in to_csv #35129

twoertwein · 2020-07-05T20:07:33Z

fixes File mode in to_csv is ignored, when passing a file object instead of a path #19827, fixes BUG: to_csv doesn't support file handles from ZipFiles #35058, fixes df.to_csv() ignores encoding when given a file object or any other filelike object. #23854 *, fixes Python 3 writing to_csv file ignores encoding argument. #13068 *, and fixes In-memory to_csv compression #22555
3 tests added
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

The first commit addresses #35058 (comment): python's open cannot take an encoding argument when mode contains a 'b' (opened in binary mode). This avoids an error when executing df.to_csv("output.csv", mode="w+b").

The second commit fixes #35058, #19827, #23854 *, and #13068 *: to_csv supports file handles in binary mode if mode contains a b and it honors encoding. to_csv re-invented a lot that was already done in get_handle. Let get_handle do the heavy lifting and remove all special cases from to_csv.

The third commit fixes #22555: some compression algorithms did not set the mode to be writeable for file handles. Together with the re-factoring in the second commit, it is now possible to write to binary file handles with compression!

*requesting an encoding for a non-binary file handles through to_csv still doesn't work but imho also doesn't make sense: specify the encoding yourself when opening the file or use a binary file handle

jreback

looks ok to me, can you add a whatsnew new in other enhancements section. (or bug fixes I/O if this is really a bug fix?)

twoertwein · 2020-07-08T16:21:42Z

thank you @jreback! The first commit is technically a bug fix as it prevents an error from occurring (df.to_csv("output.csv", mode="w+b") - to be fair there is no good reason for using binary mode when a filename is passed) but the second commit is a new feature as it allows to use binary file handles with to_csv. I will add two entries to whatsnew.

I think that this PR should also fix #23854 @eode, #19827 @colobas, and #13068 @graingert (as long as the user puts a 'b' in mode).

jreback · 2020-07-09T22:11:58Z

thank you @jreback! The first commit is technically a bug fix as it prevents an error from occurring (df.to_csv("output.csv", mode="w+b") - to be fair there is no good reason for using binary mode when a filename is passed) but the second commit is a new feature as it allows to use binary file handles with to_csv. I will add two entries to whatsnew.

I think that this PR should also fix #23854 @eode, #19827 @colobas, and #13068 @graingert (as long as the user puts a 'b' in mode).

awesome! can you have tests for each of these issues, edit the top of the PR to say you close, and add to the whatsnew?

gfyoung · 2020-07-09T22:34:09Z

This PR looks good! Let's add the tests that @jreback requested, and I think we should be good.

twoertwein · 2020-07-10T01:12:42Z

#19827: my issue #35058 is actually a duplicate of #19827 (I had a different use case but it is the same underlying technical issue). His code example is already covered in the test. I referenced his issue in the test.

#23854 and #13068: this PR will fix 'half' of both issues. If the file handle is in binary mode it will fix their issue (honor the encoding) but for non-binary file handles the issues are still not addressed. I added a test-case containing sample code from these two issues to validate the encoding for binary file handles.

~~I could try to extend this PR (or open a new one?) to cover the encoding of non-binary file handles as well (using the StringIO buffer approach for all file handles should fix this).~~ I don't think this is possible (string.encode() returns bytes but a non-binary file handle wants a string). People who provide a file handle and want their encoding to be honored should provide a binary file handle (or set the encoding themselves in the open call when creating a non-binary file handle). I changed the to_csv documentation to note that encoding isn't supported for non-binary file handles. I wouldn't mind to also mark #23854 and #13068 as fixed (one half is fixed and the other half is 'not supported').

jreback

thanks @twoertwein very comprehensive

just some clarifications and doc requests

doc string updates in to_csv looks good - could expand even to include a full example in the main docs (io.rst) and link in the doc-string

doc/source/whatsnew/v1.1.0.rst

pandas/core/generic.py

pandas/io/formats/csvs.py

pandas/io/common.py

pandas/tests/io/formats/test_to_csv.py

pandas/tests/io/test_common.py

pep8speaks · 2020-07-11T16:02:41Z

Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-08-07 00:24:21 UTC

twoertwein · 2020-07-17T18:57:55Z

I overhauled a part of to_csv that re-invented parts of get_handle (my initial PR made use of these re-inventions). The code is smaller and simpler :)

twoertwein · 2020-07-18T16:31:23Z

There seems to be an unclosed file handle on windows causing a test failure. I cannot reproduce this error on my linux machine. @jreback @gfyoung do you have an idea how to debug that? edit: it seems that at least one merged PR triggers the same failure. edit: seemed to be an CI hiccup, it passes now :)

This test has some unclosed file warnings, but other PRs have the same warnings (so hopefully not related to this PR).

~~Except the above test failure,~~ I think/hope this PR is done :)

twoertwein · 2020-07-29T15:20:25Z

rebased and 1.1 -> 1.2

@jreback @gfyoung

gfyoung · 2020-07-29T16:10:45Z

@twoertwein : It looks like there are test failures unrelated to your PR. You may need to rebase or merge master again.

twoertwein · 2020-07-29T17:25:03Z

@gfyoung rebased. Master seems to have the same issue, all other tests pass :)

gfyoung · 2020-07-29T18:05:21Z

@twoertwein : The changes look okay to me right now. Unfortunately, we will need to wait until the master failure is fixed.

jreback

@twoertwein very nice. minor test comments and a doc request. pls merge master and ping on green.

pandas/tests/io/formats/test_to_csv.py

pandas/tests/io/test_common.py

pandas/tests/io/test_compression.py

jreback · 2020-08-03T23:44:31Z

doc/source/whatsnew/v1.2.0.rst

@@ -13,6 +13,25 @@ including other versions of pandas.
 Enhancements
 ~~~~~~~~~~~~

+.. _whatsnew_120.binary_handle_to_csv:
+
+Support for binary file handles in ``to_csv``


can you add this example to the io.rst section as well

I added a new subsection in "Data Handling". Is that an appropriate place?

jreback · 2020-08-03T23:45:19Z

@gfyoung if any other comments.

pandas/core/generic.py

jreback

minor comment, pls merge master and ping on green.

doc/source/user_guide/io.rst

twoertwein · 2020-08-07T01:06:12Z

@jreback ping

jreback · 2020-08-07T11:53:17Z

thanks @twoertwein very nice!

twoertwein · 2020-08-07T12:45:09Z

@jreback and @gfyoung Thank you very much for your help getting this PR merged :)

twoertwein · 2020-08-07T17:19:11Z

pandas/core/generic.py

@@ -3080,6 +3086,10 @@ def to_csv(
               supported for compression modes 'gzip' and 'bz2'
               as well as 'zip'.

+            .. versionchanged:: 1.2.0
+
+                Compression is supported for non-binary file objects.


should have been "Compression is supported for binary file objects."! @jreback Should I create a one-line PR for this?

I opened a PR: #35615

twoertwein changed the title ~~allow to_csv to be used with binary mode~~ support binary file handles in to_csv Jul 6, 2020

jreback requested a review from gfyoung July 8, 2020 15:41

jreback added IO CSV read_csv, to_csv IO Data IO issues that don't fit into a more specific label labels Jul 8, 2020

jreback requested changes Jul 8, 2020

View reviewed changes

twoertwein requested a review from jreback July 11, 2020 03:04

jreback requested changes Jul 11, 2020

View reviewed changes

twoertwein requested a review from jreback July 14, 2020 03:21

AndrewRook mentioned this pull request Jul 18, 2020

pandas results serializer proposal PrefectHQ/prefect#2917

Closed

jreback added this to the 1.2 milestone Aug 3, 2020

jreback requested changes Aug 3, 2020

View reviewed changes

gfyoung reviewed Aug 4, 2020

View reviewed changes

pandas/core/generic.py Outdated Show resolved Hide resolved

jreback requested changes Aug 7, 2020

View reviewed changes

doc/source/user_guide/io.rst Show resolved Hide resolved

twoertwein added 3 commits August 6, 2020 20:19

io/common: no encoding when opening file in binary mode

56e5db2

to_csv: support binary file handles and their encoding

11c3fc4

to_csv: support compression for binary file handles

5eaf276

jreback approved these changes Aug 7, 2020

View reviewed changes

jreback merged commit 3b88446 into pandas-dev:master Aug 7, 2020

twoertwein commented Aug 7, 2020

View reviewed changes

This was referenced Aug 7, 2020

DOC: compression support for file objects in to_csv #35615

Merged

to_csv to Google Cloud Storage ignores compression #35677

Closed

JasonSanchez mentioned this pull request Aug 24, 2020

Read and write compressed CSVs to S3 aws/aws-sdk-pandas#359

Closed

amotl mentioned this pull request Oct 20, 2020

BUG: Problem using to_csv with BytesIO #37292

Closed

3 tasks

simonjayhawkins mentioned this pull request Dec 27, 2020

REGR: to_csv problems with zip compression and large dataframes #38714

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support binary file handles in to_csv #35129

support binary file handles in to_csv #35129

twoertwein commented Jul 5, 2020 •

edited

Loading

jreback left a comment

twoertwein commented Jul 8, 2020 •

edited

Loading

jreback commented Jul 9, 2020

gfyoung commented Jul 9, 2020

twoertwein commented Jul 10, 2020 •

edited

Loading

jreback left a comment

pep8speaks commented Jul 11, 2020 •

edited

Loading

twoertwein commented Jul 17, 2020 •

edited

Loading

twoertwein commented Jul 18, 2020 •

edited

Loading

twoertwein commented Jul 29, 2020

gfyoung commented Jul 29, 2020

twoertwein commented Jul 29, 2020

gfyoung commented Jul 29, 2020

jreback left a comment

jreback Aug 3, 2020

twoertwein Aug 4, 2020

jreback commented Aug 3, 2020

jreback left a comment

twoertwein commented Aug 7, 2020

jreback commented Aug 7, 2020

twoertwein commented Aug 7, 2020

twoertwein Aug 7, 2020

twoertwein Aug 7, 2020

support binary file handles in to_csv #35129

support binary file handles in to_csv #35129

Conversation

twoertwein commented Jul 5, 2020 • edited Loading

jreback left a comment

Choose a reason for hiding this comment

twoertwein commented Jul 8, 2020 • edited Loading

jreback commented Jul 9, 2020

gfyoung commented Jul 9, 2020

twoertwein commented Jul 10, 2020 • edited Loading

jreback left a comment

Choose a reason for hiding this comment

pep8speaks commented Jul 11, 2020 • edited Loading

Comment last updated at 2020-08-07 00:24:21 UTC

twoertwein commented Jul 17, 2020 • edited Loading

twoertwein commented Jul 18, 2020 • edited Loading

twoertwein commented Jul 29, 2020

gfyoung commented Jul 29, 2020

twoertwein commented Jul 29, 2020

gfyoung commented Jul 29, 2020

jreback left a comment

Choose a reason for hiding this comment

jreback Aug 3, 2020

Choose a reason for hiding this comment

twoertwein Aug 4, 2020

Choose a reason for hiding this comment

jreback commented Aug 3, 2020

jreback left a comment

Choose a reason for hiding this comment

twoertwein commented Aug 7, 2020

jreback commented Aug 7, 2020

twoertwein commented Aug 7, 2020

twoertwein Aug 7, 2020

Choose a reason for hiding this comment

twoertwein Aug 7, 2020

Choose a reason for hiding this comment

twoertwein commented Jul 5, 2020 •

edited

Loading

twoertwein commented Jul 8, 2020 •

edited

Loading

twoertwein commented Jul 10, 2020 •

edited

Loading

pep8speaks commented Jul 11, 2020 •

edited

Loading

twoertwein commented Jul 17, 2020 •

edited

Loading

twoertwein commented Jul 18, 2020 •

edited

Loading