support binary file handles in to_csv #35129

twoertwein · 2020-07-05T20:07:33Z

fixes File mode in to_csv is ignored, when passing a file object instead of a path #19827, fixes BUG: to_csv doesn't support file handles from ZipFiles #35058, fixes df.to_csv() ignores encoding when given a file object or any other filelike object. #23854 *, fixes Python 3 writing to_csv file ignores encoding argument. #13068 *, and fixes In-memory to_csv compression #22555
3 tests added
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

The first commit addresses #35058 (comment): python's open cannot take an encoding argument when mode contains a 'b' (opened in binary mode). This avoids an error when executing df.to_csv("output.csv", mode="w+b").

The second commit fixes #35058, #19827, #23854 *, and #13068 *: to_csv supports file handles in binary mode if mode contains a b and it honors encoding. to_csv re-invented a lot that was already done in get_handle. Let get_handle do the heavy lifting and remove all special cases from to_csv.

The third commit fixes #22555: some compression algorithms did not set the mode to be writeable for file handles. Together with the re-factoring in the second commit, it is now possible to write to binary file handles with compression!

*requesting an encoding for a non-binary file handles through to_csv still doesn't work but imho also doesn't make sense: specify the encoding yourself when opening the file or use a binary file handle

jreback

looks ok to me, can you add a whatsnew new in other enhancements section. (or bug fixes I/O if this is really a bug fix?)

twoertwein · 2020-07-08T16:21:42Z

thank you @jreback! The first commit is technically a bug fix as it prevents an error from occurring (df.to_csv("output.csv", mode="w+b") - to be fair there is no good reason for using binary mode when a filename is passed) but the second commit is a new feature as it allows to use binary file handles with to_csv. I will add two entries to whatsnew.

I think that this PR should also fix #23854 @eode, #19827 @colobas, and #13068 @graingert (as long as the user puts a 'b' in mode).

jreback · 2020-07-09T22:11:58Z

thank you @jreback! The first commit is technically a bug fix as it prevents an error from occurring (df.to_csv("output.csv", mode="w+b") - to be fair there is no good reason for using binary mode when a filename is passed) but the second commit is a new feature as it allows to use binary file handles with to_csv. I will add two entries to whatsnew.

I think that this PR should also fix #23854 @eode, #19827 @colobas, and #13068 @graingert (as long as the user puts a 'b' in mode).

awesome! can you have tests for each of these issues, edit the top of the PR to say you close, and add to the whatsnew?

gfyoung · 2020-07-09T22:34:09Z

This PR looks good! Let's add the tests that @jreback requested, and I think we should be good.

twoertwein · 2020-07-10T01:12:42Z

#19827: my issue #35058 is actually a duplicate of #19827 (I had a different use case but it is the same underlying technical issue). His code example is already covered in the test. I referenced his issue in the test.

#23854 and #13068: this PR will fix 'half' of both issues. If the file handle is in binary mode it will fix their issue (honor the encoding) but for non-binary file handles the issues are still not addressed. I added a test-case containing sample code from these two issues to validate the encoding for binary file handles.

~~I could try to extend this PR (or open a new one?) to cover the encoding of non-binary file handles as well (using the StringIO buffer approach for all file handles should fix this).~~ I don't think this is possible (string.encode() returns bytes but a non-binary file handle wants a string). People who provide a file handle and want their encoding to be honored should provide a binary file handle (or set the encoding themselves in the open call when creating a non-binary file handle). I changed the to_csv documentation to note that encoding isn't supported for non-binary file handles. I wouldn't mind to also mark #23854 and #13068 as fixed (one half is fixed and the other half is 'not supported').

jreback

thanks @twoertwein very comprehensive

just some clarifications and doc requests

doc string updates in to_csv looks good - could expand even to include a full example in the main docs (io.rst) and link in the doc-string

doc/source/whatsnew/v1.1.0.rst

pandas/core/generic.py

pandas/io/formats/csvs.py

pandas/io/common.py

pandas/tests/io/formats/test_to_csv.py

pandas/tests/io/test_common.py

pep8speaks · 2020-07-11T16:02:41Z

Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-08-07 00:24:21 UTC

twoertwein · 2020-07-17T18:57:55Z

I overhauled a part of to_csv that re-invented parts of get_handle (my initial PR made use of these re-inventions). The code is smaller and simpler :)

twoertwein · 2020-07-18T16:31:23Z

There seems to be an unclosed file handle on windows causing a test failure. I cannot reproduce this error on my linux machine. @jreback @gfyoung do you have an idea how to debug that? edit: it seems that at least one merged PR triggers the same failure. edit: seemed to be an CI hiccup, it passes now :)

This test has some unclosed file warnings, but other PRs have the same warnings (so hopefully not related to this PR).

~~Except the above test failure,~~ I think/hope this PR is done :)

twoertwein · 2020-07-29T15:20:25Z

rebased and 1.1 -> 1.2

@jreback @gfyoung

gfyoung · 2020-07-29T16:10:45Z

@twoertwein : It looks like there are test failures unrelated to your PR. You may need to rebase or merge master again.

twoertwein · 2020-07-29T17:25:03Z

@gfyoung rebased. Master seems to have the same issue, all other tests pass :)

gfyoung · 2020-07-29T18:05:21Z

@twoertwein : The changes look okay to me right now. Unfortunately, we will need to wait until the master failure is fixed.

jreback

@twoertwein very nice. minor test comments and a doc request. pls merge master and ping on green.

pandas/tests/io/formats/test_to_csv.py

pandas/tests/io/test_common.py

pandas/tests/io/test_compression.py

jreback · 2020-08-03T23:44:31Z

doc/source/whatsnew/v1.2.0.rst


+.. _whatsnew_120.binary_handle_to_csv:
+
+Support for binary file handles in ``to_csv``


can you add this example to the io.rst section as well

I added a new subsection in "Data Handling". Is that an appropriate place?

jreback · 2020-08-03T23:45:19Z

@gfyoung if any other comments.

pandas/core/generic.py

jreback

minor comment, pls merge master and ping on green.

doc/source/user_guide/io.rst

twoertwein · 2020-08-07T01:06:12Z

@jreback ping

jreback · 2020-08-07T11:53:17Z

thanks @twoertwein very nice!

twoertwein · 2020-08-07T12:45:09Z

@jreback and @gfyoung Thank you very much for your help getting this PR merged :)

twoertwein · 2020-08-07T17:19:11Z

pandas/core/generic.py


+            .. versionchanged:: 1.2.0
+
+                Compression is supported for non-binary file objects.


should have been "Compression is supported for binary file objects."! @jreback Should I create a one-line PR for this?

I opened a PR: #35615

twoertwein changed the title ~~allow to_csv to be used with binary mode~~ support binary file handles in to_csv Jul 6, 2020

jreback requested a review from gfyoung July 8, 2020 15:41

jreback added IO CSV read_csv, to_csv IO Data IO issues that don't fit into a more specific label labels Jul 8, 2020

jreback requested changes Jul 8, 2020

View reviewed changes

twoertwein requested a review from jreback July 11, 2020 03:04

jreback requested changes Jul 11, 2020

View reviewed changes

twoertwein requested a review from jreback July 14, 2020 03:21

AndrewRook mentioned this pull request Jul 18, 2020

pandas results serializer proposal PrefectHQ/prefect#2917

Closed

jreback added this to the 1.2 milestone Aug 3, 2020

jreback requested changes Aug 3, 2020

View reviewed changes

gfyoung reviewed Aug 4, 2020

View reviewed changes

pandas/core/generic.py Outdated Show resolved Hide resolved

jreback requested changes Aug 7, 2020

View reviewed changes

doc/source/user_guide/io.rst Show resolved Hide resolved

twoertwein added 3 commits August 6, 2020 20:19

io/common: no encoding when opening file in binary mode

56e5db2

to_csv: support binary file handles and their encoding

11c3fc4

to_csv: support compression for binary file handles

5eaf276

jreback approved these changes Aug 7, 2020

View reviewed changes

jreback merged commit 3b88446 into pandas-dev:master Aug 7, 2020

twoertwein commented Aug 7, 2020

View reviewed changes

This was referenced Aug 7, 2020

DOC: compression support for file objects in to_csv #35615

Merged

to_csv to Google Cloud Storage ignores compression #35677

Closed

JasonSanchez mentioned this pull request Aug 24, 2020

Read and write compressed CSVs to S3 aws/aws-sdk-pandas#359

Closed

amotl mentioned this pull request Oct 20, 2020

BUG: Problem using to_csv with BytesIO #37292

Closed

3 tasks

simonjayhawkins mentioned this pull request Dec 27, 2020

REGR: to_csv problems with zip compression and large dataframes #38714

Closed


		.. _whatsnew_120.binary_handle_to_csv:

		Support for binary file handles in ``to_csv``


		.. versionchanged:: 1.2.0

		Compression is supported for non-binary file objects.

Uh oh!

support binary file handles in to_csv #35129

support binary file handles in to_csv #35129

Uh oh!

Conversation

twoertwein commented Jul 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jreback left a comment

Choose a reason for hiding this comment

Uh oh!

twoertwein commented Jul 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jreback commented Jul 9, 2020

Uh oh!

gfyoung commented Jul 9, 2020

Uh oh!

twoertwein commented Jul 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jreback left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pep8speaks commented Jul 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated at 2020-08-07 00:24:21 UTC

Uh oh!

twoertwein commented Jul 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

twoertwein commented Jul 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

twoertwein commented Jul 29, 2020

Uh oh!

gfyoung commented Jul 29, 2020

Uh oh!

twoertwein commented Jul 29, 2020

Uh oh!

gfyoung commented Jul 29, 2020

Uh oh!

jreback left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jreback Aug 3, 2020

Choose a reason for hiding this comment

Uh oh!

twoertwein Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

jreback commented Aug 3, 2020

Uh oh!

Uh oh!

jreback left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

twoertwein commented Aug 7, 2020

Uh oh!

jreback commented Aug 7, 2020

Uh oh!

twoertwein commented Aug 7, 2020

Uh oh!

twoertwein Aug 7, 2020

Choose a reason for hiding this comment

Uh oh!

twoertwein Aug 7, 2020

twoertwein commented Jul 5, 2020 •

edited

Loading

twoertwein commented Jul 8, 2020 •

edited

Loading

twoertwein commented Jul 10, 2020 •

edited

Loading

pep8speaks commented Jul 11, 2020 •

edited

Loading

twoertwein commented Jul 17, 2020 •

edited

Loading

twoertwein commented Jul 18, 2020 •

edited

Loading