Skip to content

COMPAT: Properly encode filenames in read_csv #24758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 14, 2019
Merged

COMPAT: Properly encode filenames in read_csv #24758

merged 1 commit into from
Jan 14, 2019

Conversation

@gfyoung
Copy link
Member

@gfyoung gfyoung commented Jan 13, 2019

Python 3.6+ changes the default encoding to utf8 (PEP 529), which conflicts with the encoding of Windows (mbcs). This conflicts rears its head when using only the C engine of read_csv because of the low level functionality that we implement ourselves.

This fix checks if we're using Python 3.6+ and on Windows, after which we force the encoding to mbcs

Closes #15086.

@gfyoung gfyoung added Bug Unicode Unicode strings IO CSV read_csv, to_csv 2/3 Compat Windows Windows OS Compat pandas objects compatability with Numpy or Python functions labels Jan 13, 2019
@gfyoung gfyoung added this to the 0.24.0 milestone Jan 13, 2019
@gfyoung gfyoung requested a review from jreback January 13, 2019 23:43
@gfyoung gfyoung force-pushed the py-36-windows-mbcs-csv branch from f8307fc to cfad6ec Compare January 13, 2019 23:43
@jreback
Copy link
Contributor

jreback commented Jan 13, 2019

lgtm. ping on green.

@codecov
Copy link

codecov bot commented Jan 14, 2019

Codecov Report

Merging #24758 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master   #24758   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files         166      166           
  Lines       52363    52363           
=======================================
  Hits        48376    48376           
  Misses       3987     3987
Flag Coverage Δ
#multiple 90.81% <ø> (ø) ⬆️
#single 42.91% <ø> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f160c7d...cfad6ec. Read the comment docs.

@gfyoung
COMPAT: Properly encode filenames in read_csv
ab663b4 
Python 3.6+ changes the default encoding to
UTF8 (PEP 529), which conflicts with the
encoding of Windows (MBCS).

This fix checks if we're using Python 3.6+
and on Windows, after which we force the
encoding to "mbcs".

Closes pandas-devgh-15086.
@gfyoung
Copy link
Member Author

gfyoung commented Jan 14, 2019

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=6891

@jreback : The lint failure is unrelated to my changes. However, I'll rebase and push again to be safe.

@gfyoung gfyoung force-pushed the py-36-windows-mbcs-csv branch from cfad6ec to ab663b4 Compare January 14, 2019 00:27
@pandas-dev pandas-dev deleted a comment from codecov bot Jan 14, 2019
@codecov
Copy link

codecov bot commented Jan 14, 2019

Codecov Report

Merging #24758 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master   #24758   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files         166      166           
  Lines       52363    52363           
=======================================
  Hits        48376    48376           
  Misses       3987     3987
Flag Coverage Δ
#multiple 90.81% <ø> (ø) ⬆️
#single 42.91% <ø> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 453fa85...ab663b4. Read the comment docs.

@pandas-dev pandas-dev deleted a comment from codecov bot Jan 14, 2019
@codecov
Copy link

codecov bot commented Jan 14, 2019

Codecov Report

Merging #24758 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master   #24758   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files         166      166           
  Lines       52363    52363           
=======================================
  Hits        48376    48376           
  Misses       3987     3987
Flag Coverage Δ
#multiple 90.81% <ø> (ø) ⬆️
#single 42.91% <ø> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 453fa85...ab663b4. Read the comment docs.

@gfyoung
Copy link
Member Author

gfyoung commented Jan 14, 2019

@jreback : Rebased, and all is green. PTAL.

@gfyoung gfyoung mentioned this pull request Jan 14, 2019
Closed
@jreback jreback merged commit e2bf1ff into pandas-dev:master Jan 14, 2019
@jreback
Copy link
Contributor

jreback commented Jan 14, 2019

thanks @gfyoung

@TomAugspurger
Copy link
Contributor

Thank @gfyoung. Do you know if the CI would have failed on that path without your fix? IIUC, the failure was somewhat system-dependent, though I don't recall the exact details.

@gfyoung
Copy link
Member Author

gfyoung commented Jan 14, 2019 •
edited
Loading

Do you know if the CI would have failed on that path without your fix? IIUC, the failure was somewhat system-dependent, though I don't recall the exact details.

@TomAugspurger : Yes, it would have. It's a Windows Python 3.6+ issue.

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@gfyoung @Pingviinituutti
COMPAT: Properly encode filenames in read_csv (pandas-dev#24758)
ee1be2c 
Python 3.6+ changes the default encoding to
UTF8 (PEP 529), which conflicts with the
encoding of Windows (MBCS).

This fix checks if we're using Python 3.6+
and on Windows, after which we force the
encoding to "mbcs".

Closes pandas-devgh-15086.
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@gfyoung @Pingviinituutti
COMPAT: Properly encode filenames in read_csv (pandas-dev#24758)
518ebf2 
Python 3.6+ changes the default encoding to
UTF8 (PEP 529), which conflicts with the
encoding of Windows (MBCS).

This fix checks if we're using Python 3.6+
and on Windows, after which we force the
encoding to "mbcs".

Closes pandas-devgh-15086.
@vnlitvinov vnlitvinov mentioned this pull request Mar 18, 2019
Merged
4 tasks
@gfyoung gfyoung mentioned this pull request Oct 29, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jreback jreback Awaiting requested review from jreback

Assignees

No one assigned

Labels

Bug Compat pandas objects compatability with Numpy or Python functions IO CSV read_csv, to_csv Unicode Unicode strings Windows Windows OS

Projects

None yet

Milestone

0.24.0

Development

Successfully merging this pull request may close these issues.

OSError when reading file with accents in file path

3 participants

@gfyoung @jreback @TomAugspurger