Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: s3fs pytest unraisable exception #1012

Merged
merged 63 commits into from
Nov 16, 2023
Merged
Changes from 1 commit
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
e397896
install optional fsspec backends in the CI for some python versions
lobis Oct 19, 2023
a93eb6c
Merge branch 'main' into fsspec-optional-backends
lobis Oct 19, 2023
c674789
update submit interface
lobis Oct 19, 2023
8da9c09
annotation
lobis Oct 19, 2023
3242517
do not use named arguments for the path/url
lobis Oct 19, 2023
59bec72
Future init
lobis Oct 19, 2023
2a97904
add s3fs and sshfs as test dependencies (fsspec beckends)
lobis Oct 19, 2023
181fa74
Merge remote-tracking branch 'origin/source-futures-submit' into fssp…
lobis Oct 19, 2023
e07231f
test fsspec s3 for more combination of parameters
lobis Oct 19, 2023
5fa2dcc
remove pip install from ci
lobis Oct 19, 2023
b3ddcc2
Merge branch 'main' into fsspec-optional-backends
lobis Oct 19, 2023
8a842c9
style: pre-commit fixes
pre-commit-ci[bot] Oct 19, 2023
3fea743
Merge branch 'main' into fsspec-optional-backends
lobis Oct 19, 2023
662b0f9
revert test order
lobis Oct 19, 2023
8cd55dd
remove dependencies as a test
lobis Oct 20, 2023
ae0a96e
add s3fs to test
lobis Oct 20, 2023
ba58b8f
exclude s3fs to python version 3.12
lobis Oct 20, 2023
3b7dfeb
add sshfs test (skipped)
lobis Oct 20, 2023
a60ca28
fix pytest
lobis Oct 20, 2023
cbbd07d
asyncio not available in 3.12
lobis Oct 20, 2023
06bbf92
asyncio not available in 3.12
lobis Oct 20, 2023
e9c71ac
add comment for fsspec threads
lobis Oct 20, 2023
37e5a24
attempt to close resources
lobis Oct 23, 2023
d2fb8a4
handle s3fs case separate for now
lobis Oct 23, 2023
d30e531
attempt to pass tests
lobis Oct 23, 2023
ee5eeab
attempt to pass tests
lobis Oct 23, 2023
993f979
simplified
lobis Oct 23, 2023
26924e2
remove support for use_threads option, run non-async fs in threads us…
lobis Oct 24, 2023
d898dbc
stop the loop on resource shutdown
lobis Oct 24, 2023
58fe0f0
add skip for xrootd due to server issues
lobis Oct 24, 2023
a8550eb
Merge remote-tracking branch 'origin/main' into fsspec-optional-backends
lobis Oct 25, 2023
09aa93a
remove skip for xrootd
lobis Oct 25, 2023
2e200b5
remove shutdown
lobis Oct 25, 2023
45fddff
Merge branch 'main' into fsspec-optional-backends
lobis Oct 25, 2023
2f73c45
merge and fix conflicts
lobis Oct 26, 2023
27ca078
understand ci fail
lobis Oct 30, 2023
85ec20f
enable test that uses s3fs
lobis Oct 31, 2023
a51630d
Merge branch 'main' into s3fs-pytest-issue
lobis Oct 31, 2023
e0054d1
Merge remote-tracking branch 'origin/main' into s3fs-pytest-issue
lobis Nov 3, 2023
45b9f4e
Merge branch 'main' into s3fs-pytest-issue
lobis Nov 3, 2023
92e5999
merge origin/main
lobis Nov 15, 2023
484a7bf
Merge remote-tracking branch 'origin/s3fs-pytest-issue' into s3fs-pyt…
lobis Nov 15, 2023
36dac91
skip test for CI
lobis Nov 15, 2023
a9fcdbe
merge with main
lobis Nov 16, 2023
8b92a4e
skip tests on problematic python versions
lobis Nov 16, 2023
8534658
test S3Source only in its test file
lobis Nov 16, 2023
d322aa2
fix pytest parameter
lobis Nov 16, 2023
6a17ac7
rename test
lobis Nov 16, 2023
827b340
rename test function (duplicate name)
lobis Nov 16, 2023
7bce9bb
imports
lobis Nov 16, 2023
e604dbb
rerun on timeout
lobis Nov 16, 2023
85afac3
rerun on timeout
lobis Nov 16, 2023
6e79951
remove minio
lobis Nov 16, 2023
56e101d
skip s3fs test instead
lobis Nov 16, 2023
f3f1fa4
rename test
lobis Nov 16, 2023
a7ae473
no s3 in test names
lobis Nov 16, 2023
f014491
improve pytest regex
lobis Nov 16, 2023
3fa3ae8
python 3.8 skip
lobis Nov 16, 2023
5b23324
skip s3_fail when timeout
lobis Nov 16, 2023
4f7113f
error tuple
lobis Nov 16, 2023
aa8395a
Merge branch 'main' into s3fs-pytest-issue
lobis Nov 16, 2023
26e7108
line jump
lobis Nov 16, 2023
fb504d8
Merge branch 'main' into s3fs-pytest-issue
jpivarski Nov 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
attempt to close resources
lobis committed Oct 23, 2023
commit 37e5a2443689433c6c095b93275915ce56ec1d02
15 changes: 13 additions & 2 deletions src/uproot/source/fsspec.py
Original file line number Diff line number Diff line change
@@ -50,17 +50,24 @@ def __init__(self, file_path: str, **options):
# This should never be triggered unless the user explicitly set the `use_async` flag for a non-async backend
raise ValueError(f"Filesystem {protocol} does not support async")

self._fs, self._file_path = fsspec.core.url_to_fs(file_path, **storage_options)

if not self._use_threads:
self._executor = uproot.source.futures.TrivialExecutor()
elif self._use_async:
self._executor = FSSpecLoopExecutor(fsspec.asyn.get_loop())

async def get_session():
return await self._fs.set_session()

self._executor.session = asyncio.run_coroutine_threadsafe(
get_session(), self._executor.loop
).result()
else:
self._executor = concurrent.futures.ThreadPoolExecutor(
max_workers=self._num_workers
)

self._fs, self._file_path = fsspec.core.url_to_fs(file_path, **storage_options)

if not self._use_threads:
# assert threading.active_count() == 1
...
@@ -197,6 +204,7 @@ def closed(self) -> bool:
class FSSpecLoopExecutor(uproot.source.futures.Executor):
def __init__(self, loop: asyncio.AbstractEventLoop):
self.loop = loop
self.session = None

def submit(self, coroutine, /, *args, **kwargs) -> concurrent.futures.Future:
if not asyncio.iscoroutinefunction(coroutine):
@@ -205,3 +213,6 @@ def submit(self, coroutine, /, *args, **kwargs) -> concurrent.futures.Future:
raise RuntimeError("cannot submit coroutine while loop is not running")
coroutine_object = coroutine(*args, **kwargs)
return asyncio.run_coroutine_threadsafe(coroutine_object, self.loop)

def shutdown(self, wait: bool = True):
asyncio.run_coroutine_threadsafe(self.session.close(), self.loop).result()