-
Notifications
You must be signed in to change notification settings - Fork 193
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Petl use fsspec #494
Petl use fsspec #494
Conversation
use fsspec for handling remote protocols Add RemoteSource for handling all remote sources with fsspec Remove S3Source replaced by fsspec Keep SMBSource as it isn't handled by fsspec Refactored namespaces to remotes
remove codec implementation in petl
Hi @juarezr, looking good so far. Please feel free to request a review whenever you're ready. |
Oh! Sorry! I forgot to request your review. I think it's ready to review and merge after:
Can you review it, please? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @juarezr, a couple of minor comments.
One slightly bigger question, currently this does not support local files or http(s) URLs with zstd, lz4 or lzma (xz) compression, because these are not opened via fsspec, is that correct?
Ready to merge. Co-authored-by: Alistair Miles <alimanfoo@googlemail.com>
If using If using |
I see, thanks. Would it be possible to use fsspec for all values except those with 'smb://' protocol? If so, then all compressors would be supported for local files without having to add the 'file://' protocol. I don't think there is any issue with compatibility, i.e., reading '/path/to/file.csv.gz' should still behave exactly the same as before, it just gets opened via fsspec. |
Btw I noticed that fsspec does not support Python 2, but petl can drop support for Python 2 also, it is EOL. I realise that dropping Python 2 support may take some work, however, and so might be too much to deal with in this PR. |
Also PIP is hammering another nail in the python2's coffin. It's planned for in pip 21.0 (Jan 2021). But I don't know how this impacts existing python2 packages.
For doing this we need a little change: prefix raw paths with Doing this will also makes I think that supporting Because of this decoupling, I kept in this PR:
It's possible to ressurect the
I agree. BTW, I started digging in a |
Thanks @juarezr. I'm happy to follow what you think is the best approach here. |
Thans @alimanfoo ! |
Well I speak only one language, and even that badly, so I am in awe of anyone who can do technical communication in a second language 😄 |
Ready to merge? |
If you don't have any objections, I think it's ready to merge. |
Changes
fsspec
.SMBHandler
class asfsspec
does not handle it yet.Checklist