-
Notifications
You must be signed in to change notification settings - Fork 42
Description
When #152 is merged, user subclasses that override upath.core._FSSpecAccessor
will very likely not work correctly with Python3.12, because the 3.12 implementation doesn't use the accessor.
There should be a recommended way for users to subclass UPath
for Python3.12 AND Python3.8-Python3.11 without having to maintain two code paths.
Searching GitHub, I could collect an example set of user subclasses that can be split into 3 4 intended use cases:
(1) customize fsspec filesystem creation
here the user wants to change the way the fsspec filesystem instance is created:
- GitHubAccessor which gets token kwarg from environment: https://github.com/juftin/textual-universal-directorytree/blob/110770f2ee40ab5afff7eade635caad644d80848/textual_universal_directorytree/alternate_paths.py#L15-L27
- HTTPAccessor that wraps the fsspec filesystem with a cache: https://github.com/joouha/euporie/blob/620ce11442e5f7d189966950afa3c088b5d0a41f/euporie/core/path.py#L59-L77
- AirflowCloudAccessor which completely customizes fsspec instance creation: https://github.com/apache/airflow/blob/07fd3646fa765859a152731bf135ea944d0753d7/airflow/io/path.py#L46-L68
upath.UPath
should offer a classmethod _make_fs()
(?) to allow fsspec filesystem instance customization.
It should be possible too to support old code on python3.12 by detecting when the _default_accessor
attribute is present and use that to make a compatible _make_fs()
(?) fallback method.
(2) customize path formatting
here the user wants to change the way the path is formatted. Examples:
- GitHubAccessor which removes the leading slash: https://github.com/phil65/mknodes/blob/0cd6671fe05d66ad44bd17e5287ac8534eccea61/mknodes/utils/pathhelpers.py#L22-L27
- DataAccessor which returns the full URI: https://github.com/joouha/euporie/blob/620ce11442e5f7d189966950afa3c088b5d0a41f/euporie/core/path.py#L110-L113
(3) fix behavior of some accessor methods
here the behavior of a fsspec filesystem method is fixed by fixing the accessor method. Example:
- BoxAccessor wich fixes an issue with
boxfs.mkdir
: https://github.com/IBM/boxfs/blob/718fb0071d20a7004f44fe2fa0eac26dc9c3d5d5/src/boxfs/_upath.py#L4-L12
(4) customize storage_options extraction
Not really specific to the Accessor subclassing, because the implementation is overriding __new__
in the UPath subclass. The intention is to extract additional fsspec storage_options from the provided args when instantiating UPath()
. Example:
ObjectStoragePath
extracts a connection id from the urlpath: https://github.com/apache/airflow/blob/07fd3646fa765859a152731bf135ea944d0753d7/airflow/io/path.py#L96-L153
TODO
While this list in not complete, it does cover 3 4 reasonably easy to solve cases for making the transition to 3.12 less bumpy for custom subclasses.
- (ALL) detect custom
_accessor
inpy3.12
classes and provide warning with instructions on how to migrate. - (1) provide customizable fsspec filesystem factory classmethod
- (2) provide instructions for how to migrate to
.path
(with potential customization) - (3) recommend overriding
UPath.mkdir
, etc... directly. - (4) provide customizable fsspec storage_option extraction classmethod