-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error when writing to a CloudPath file using Pandas #242
Comments
Hi @jlondonob, this looks to be a specific case of the general problem captured by Unfortunately, as described in #128, it's a known limitation that cloud paths do not behave as expected when called with Thank you for some discussion in your report, and you're welcome to continue contributing to the discussion in #128. I'm going to close this issue for now in order to centralize the discussion there. I also want to note that the workaround that you found of calling |
Hello, Just to flag it here, there is the same issue with |
Context
I'm getting an error when I try to write files to the cloud using
pandas
andcloudpathlib
.Code
Error
My current solution is to call the method
as_uri()
when writing objects, but this syntax is not verypathlib
like:Problem
Prior to writing a file,
pandas
tries to turnPathLike
objects to strings using their__fspath__()
method (seestringify_path
function). ForCloudPath
objects, the__fspath__()
method is set to return the path of the local cache instead of the cloud URI (this was introduced with 008663f, see discussion in #72).This behavior makes
pandas
try to write files as if they were local, instead of uploading them to the cloud. This can lead to two outcomes:pandas
will raise an error since the local version of the parent path won't exist.Both of these outcomes are undesirable for the use case presented above.
Discussion
__fspath()__
method, but was later dropped on the basis of PEP 519.s3Path
issue.pandas
users to be able to write objects as if working withpathlib
. Do you think it would be possible to find a solution to this?The text was updated successfully, but these errors were encountered: