Python 3.10 alpha 7 introduces new changes to pathlib

[alexdelorenzo]

The following works on alpha 6 and below, but alpha 7 introduces this error:

In [1]: from aiopath import AsyncPath                                                                                                                                  
                                                                                                                                                                       
In [2]: home = await AsyncPath.home()                                                                                                                                  
---------------------------------------------------------------------------                                                                                            
TypeError                                 Traceback (most recent call last)                                                                                            
<ipython-input-2-ef16332cd56b> in <module>                                                                                                                             
----> 1 home = await AsyncPath.home()                                                                                                                                  
                                                                                                                                                                       
~/.pyenv/versions/3.10-dev/lib/python3.10/site-packages/aiopath/path.py in home(cls)                                                                                   
    391     returned by os.path.expanduser('~')).                                                                                                                      
    392     """                                                                                                                                                        
--> 393     coro = cls()._flavour.gethomedir(None)                                                                                                                     
    394     homedir: str = await coro                                                                                                                                  
    395                                                                                                                                                                
                                                                                                                                                                       
~/.pyenv/versions/3.10-dev/lib/python3.10/site-packages/aiopath/path.py in __new__(cls, *args, **kwargs)                                                               
    133       cls = AsyncWindowsPath if os.name == 'nt' else AsyncPosixPath                                                                                            
    134 
--> 135     self = cls._from_parts(args, init=False)
    136 
    137     if not self._flavour.is_supported:

TypeError: PurePath._from_parts() got an unexpected keyword argument 'init'

[alexdelorenzo]

The specific error in this ticket is fixed in this branch: https://github.com/alexdelorenzo/aiopath/tree/Python-3.10

Versions 0.6.x on PyPI work with Python 3.10a7: python3 -m install aiopath==0.6.1

[alexdelorenzo]

Using the latest branch of pytest (via https://github.com/pytest-dev/pytest) with 3.10 support, it appears that the coroutine method AsyncPath.resolve() is broken.

______________________________________________________________________________ test_home ______________________________________________________________________________                                                                           
                                                                                                                                                                                                                                                  
    @pytest.mark.asyncio                                                                                                                                                                                                                          
    async def test_home():                                                                                                                                                                                                                        
      home = Path.home()                                                                                                                                                                                                                          
      ahome = await AsyncPath.home()                                                                                                                                                                                                              
                                                                                                                                                                                                                                                  
>     await _test_is(home, ahome)                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                  
tests.py:91:                                                                                                                                                                                                                                      
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                                           
tests.py:67: in _test_is                                                                                                                                                                                                                          
    await _test_is(path.parent, apath.parent, test_parent=False)                                                                                                                                                                                  
tests.py:74: in _test_is                                                                                                                                                                                                                          
    await apath.resolve(),                                                                                                                                                                                                                        
aiopath/path.py:491: in resolve                                                                                                                                                                                                                   
    s: Optional[str] = await self._flavour.resolve(self, strict=strict)                                                                                                                                                                           
aiopath/flavours.py:90: in resolve                                                                                                                                                                                                                
    result = await _resolve(base, str(path))                                                                                                                                                                                                      
aiopath/flavours.py:73: in _resolve                                                                                                                                                                                                               
    target = await accessor.readlink(newpath)                                                                                                                                                                                                     
../../../.pyenv/versions/3.10-dev/lib/python3.10/site-packages/aiofiles/os.py:13: in run                                                                                                                                                          
    return await loop.run_in_executor(executor, pfunc)                                                                                                                                                                                            
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                                           
                                                                                                                                                                                                                                                  
self = None                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                  
    def run(self):                                                                                                                                                                                                                                
        if not self.future.set_running_or_notify_cancel():                                                                                                                                                                                        
            return                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                  
        try:                                                                                                                                                                                                                                      
>           result = self.fn(*self.args, **self.kwargs)                                                                                                                                                                                           
E           TypeError: readlink() takes exactly 1 positional argument (2 given)                                                                                                                                                                   
                                                                                                                                                                                                                                                  
../../../.pyenv/versions/3.10-dev/lib/python3.10/concurrent/futures/thread.py:52: TypeError

[alexdelorenzo]

Here is a list of changes to the upstream pathlib package from the 3.10 alpha 7 stdlib:

• bpo-39924: handle missing os functions more consistently in pathlib python/cpython#19220
• bpo-43510: Implement PEP 597 opt-in EncodingWarning. python/cpython#19481
• bpo-39895: Move pathlib.Path.touch() implementation into the path accessor python/cpython#18838
• bpo-40038: pathlib: remove partial support for preserving accessor when modifying a path python/cpython#19342
• bpo-39906: add follow_symlinks parameter to pathlib.Path.stat() and chmod() python/cpython#18864
• bpo-42999: expand and clarify pathlib.Path.link_to() documentation. python/cpython#24294
• bpo-39659: Route calls from pathlib.Path to os.getcwd() via the path accessor python/cpython#18834
• bpo-39899: Make pathlib use os.path.expanduser() to expand home directories python/cpython#18841
• bpo-40107: stop using os.open() to implement pathlib.Path.open() python/cpython#25240

Full list of changes here: https://github.com/python/cpython/commits/master/Lib/pathlib.py

These changes will need to be ported to aiopath's Python-3.10 branch.

[alexdelorenzo]

Just some notes for myself:

I started this project as a fork of pathlib for an opportunity to port synchronous code to asynchronous Python and to work with some standard library code. I wanted to keep the same class hierarchy and API to address use cases that might consume those interfaces, even if I personally don't plan on using them.

It might be worth profiling trio's approach of just wrapping synchronous I/O pathlib methods with decorators that submit the methods to a threadpool in the background: https://github.com/python-trio/trio/blob/6754c74eacfad9cc5c92d5c24727a2f3b620624e/trio/_path.py

That way I'm not beholden to upstream maintainers' changes in the stdlib, and as long as the API stays the same, aiopath won't have to change much if at all between changes to the stdlib.

The downside is punting some CPU-bound code to the threadpool, which, all in all, probably has an insignificant overhead.

It can also introduce problems like these:

• https://github.com/python-trio/trio/issues?q=is%3Aissue+pathlib+is%3Aopen
• https://github.com/python-trio/trio/issues?q=is%3Aissue+pathlib+is%3Aclosed

However, Barney Gale's updates to pathlib that appeared in 3.10 alpha 7 are stepping stones on a roadmap towards an extensible pathlib: https://discuss.python.org/t/make-pathlib-extensible/3428/

In a world with an extensible pathlib, it makes sense for aiopath to have the current class hierarchy it has (with AsyncPath subclassing AsyncPurePath etc). I'd also assume in such a world, pathlib would also have a stable interface to support consumers like aiopath.

[barneygale]

I'm sorry my changes have broken things! As always with extending private APIs in the standard library, it can be a bit of a minefield. As you mentioned, I'm working towards a pathlib.AbstractPath class which supports subclassing. I'm hoping this can land in Python 3.11, all being well :)