Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG-REPORT] Hitting pybind11::handle::inc_ref() when using NumPy 1.26 with vaex 4.18.0 #2439

Open
setu4993 opened this issue Oct 2, 2024 · 10 comments
Open

[BUG-REPORT] Hitting pybind11::handle::inc_ref() when using NumPy 1.26 with vaex 4.18.0 #2439

setu4993 opened this issue Oct 2, 2024 · 10 comments

Comments

@setu4993
Copy link

setu4993 commented Oct 2, 2024

Description

All our unit tests and validations which worked fine on v4.17.1 started failing upon updating to v4.18.0. Curiously, this is occurring only on GitHub Actions so far (ubuntu-latest images) and not reproducible on M-series MacBooks. I suspect this is limited to some instruction sets.

I don't have a great sample right now (but can get one tomorrow) to reproduce this, but the CI run here might be helpful to compare the diff. This is occurring across repos but this public one was the easiest to link to.

Software information

  • Vaex version (import vaex; vaex.__version__): 4.18.0
  • Vaex was installed via: pip / conda-forge / from source: pip
  • OS: GitHub Actions, ubuntu-latest
@iisakkirotko
Copy link

iisakkirotko commented Oct 2, 2024 •
edited
Loading

I think we're running into the same issue in this run at widgetti/solara. We also see an error on windows: Windows fatal exception: access violation resulting in a seg fault.

The full log for Windows 
Thread 0x00000694 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\hash.py", line 171 in add
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\cpu.py", line 344 in process
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\execution.py", line 564 in process_tasks
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\execution.py", line 500 in process_part
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\multithreading.py", line 80 in wrapped
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\concurrent\futures\thread.py", line 58 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\concurrent\futures\thread.py", line 83 in _worker
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 917 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Current thread 0x00000b14 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\hash.py", line 171 in add
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\cpu.py", line 344 in process
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\execution.py", line 564 in process_tasks
  File "Windows fatal exception: C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\execution.pyaccess violation"

, line 500 in process_part
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\multithreading.py", line 80 in wrapped
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\concurrent\futures\thread.py", line 58 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\concurrent\futures\thread.py", line 83 in _worker
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 917 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001810 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 316 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 581 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 1304 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001468 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x000004cc (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line [38](https://github.com/widgetti/solara/actions/runs/11138736498/job/30955789237#step:9:39)0 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000a44 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x000019ec (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x0000130c (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000798 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000b18 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001ff0 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001074 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000958 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001a84 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000f34 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x000018e4 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001b6c (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x000013e4 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000e08 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000c94 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x000009b0 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001514 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001454 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001220 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000614 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001a6c (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000c18 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000c1c (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x000015f8 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x000006ec (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x000002d0 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001058 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001510 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001a40 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000334 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001758 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\queue.py", line 171 in get
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 379 in dispatch_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 213 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00000d24 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 326 in read_directory_changes
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\winapi.py", line 380 in read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 67 in _read_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\read_directory_changes.py", line 70 in queue_events
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\watchdog\observers\api.py", line 158 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 306 in _WidgetContextAwareThread__bootstrap
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\patch.py", line 284 in WidgetContextAwareThread__bootstrap

Thread 0x00001de4 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 581 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\IPython\core\history.py", line 894 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\IPython\core\history.py", line 60 in only_when_enabled
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\decorator.py", line 232 in fun
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap

Thread 0x00000468 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\windows_events.py", line 788 in _poll
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\windows_events.py", line 4[39](https://github.com/widgetti/solara/actions/runs/11138736498/job/30955789237#step:9:40) in select
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\base_events.py", line 1869 in _run_once
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\base_events.py", line 601 in run_forever
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\asyncio\windows_events.py", line 321 in run_forever
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\server\kernel_context.py", line 290 in _run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 917 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap

Thread 0x0000095c (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\concurrent\futures\_base.py", line 4[41](https://github.com/widgetti/solara/actions/runs/11138736498/job/30955789237#step:9:42) in result
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\concurrent\futures\_base.py", line 609 in result_iterator
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\itertools.py", line 5 in buffer
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\multithreading.py", line 104 in map
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\execution.py", line 432 in execute_generator
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\execution.py", line 308 in execute
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\dataframe.py", line [42](https://github.com/widgetti/solara/actions/runs/11138736498/job/30955789237#step:9:43)1 in execute
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\dataframe.py", line 1780 in _delay
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\vaex\dataframe.py", line 7194 in groupby
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\lab\utils\dataframe.py", line 63 in df_value_count
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\solara\components\cross_filter.py", line 82 in CrossFilterSelect
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 1707 in _render
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 2180 in _visit_children_values
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 2183 in <listcomp>
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 2183 in _visit_children_values
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 2188 in <dictcomp>
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 2188 in _visit_children_values
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 2175 in _visit_children
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 1625 in _render
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 1742 in _render
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 1742 in _render
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 1[43](https://github.com/widgetti/solara/actions/runs/11138736498/job/30955789237#step:9:44)6 in render
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\reacton\core.py", line 2210 in render
  File "D:\a\solara\solara\tests\unit\cross_filter_component_test.py", line 36 in test_cross_filter_select
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\python.py", line 159 in pytest_pyfunc_call
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\python.py", line 1627 in runtest
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 174 in pytest_runtest_call
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line [51](https://github.com/widgetti/solara/actions/runs/11138736498/job/30955789237#step:9:52)3 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 242 in <lambda>
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 341 in from_call
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 241 in call_and_report
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 132 in runtestprotocol
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 113 in pytest_runtest_protocol
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 362 in pytest_runtestloop
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 337 in _main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 283 in wrap_session
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 330 in pytest_cmdline_main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\config\__init__.py", line 175 in main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\config\__init__.py", line 201 in console_main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\Scripts\pytest.exe\__main__.py", line 7 in <module>
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\runpy.py", line 87 in _run_code
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\runpy.py", line 197 in _run_module_as_main
D:\a\_temp\691c007b-3684-4dbd-aff5-fd213c6997ea.sh: line 3:   9[56](https://github.com/widgetti/solara/actions/runs/11138736498/job/30955789237#step:9:57) Segmentation fault      pytest tests/unit --doctest-modules --timeout=60
  • Vaex version: 4.18.0
  • Vaex was installed via: pip / conda-forge / from source: pip
  • OS: GitHub Actions, ubuntu-latest and windows-latest

@maartenbreddels
Copy link
Member

The relevant part of the logs:

self = <vaex.hash.HashMapUnique object at 0x7fe128022ee0>

    def flatten(self):
        if self.dtype == object:
            return self # already flat
>       keys = self._internal.key_array()
E       RuntimeError: pybind11::handle::inc_ref() PyGILState_Check() failure.

/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/vaex/hash.py:78: RuntimeError
----------------------------- Captured stderr call -----------------------------
pybind11::handle::inc_ref() is being called while the GIL is either not held or invalid. Please see https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors for debugging advice.
The failing pybind11::handle::inc_ref() call was triggered on a numpy.ndarray object.

I think this is a bug in vaex-core which only gets exposed now due to upgrading pybind11. We should write a test to expose this, and see if we can reproduce this in CI. I think we should yank the vaex-core release to avoid other people hitting this.

@setu4993 setu4993 mentioned this issue Oct 2, 2024
Closed
2 tasks
@setu4993
Copy link
Author

setu4993 commented Oct 2, 2024

+1, yanking seems like a good idea if this is arising from vaex-core.

@maartenbreddels
Copy link
Member

it's yanked

@maartenbreddels
Copy link
Member

I could not reproduce this on a local ubuntu machine with Python 3.11, but i'll continue to see if I can reproduce it (help is welcome)

@ddelange
Copy link
Contributor

ddelange commented Oct 7, 2024 •
edited
Loading

I googled RuntimeError: pybind11::handle::inc_ref() PyGILState_Check() failure and blamed it onto the following release from 2022 (3rd bullet point):

https://github.com/pybind/pybind11/releases/tag/v2.10.2

looks like the end-user can disable these checks using environment variables although it does look like a valid point made in the PR description

@maartenbreddels
Copy link
Member

Update:
We do not build with -DNDEBUG when we use cibuildwheel, but in a regular build we do pass this to the compiler. This causes our release to be shipped with this check, while in our CI we do not have this check.
This is the exact opposite of what we want.

I'm not sure why this is, and who sets this (layers of abstraction are not always great...)

@maartenbreddels maartenbreddels mentioned this issue Oct 7, 2024
Merged
@maartenbreddels
Copy link
Member

See #2440

@ddelange
Copy link
Contributor

ddelange commented Oct 9, 2024

https://pypi.org/project/vaex-core/4.18.1/

@setu4993
Copy link
Author

setu4993 commented Oct 9, 2024

Thank you!

@iisakkirotko iisakkirotko mentioned this issue Oct 9, 2024
Open
@franz101 franz101 mentioned this issue Oct 10, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@maartenbreddels @setu4993 @iisakkirotko @ddelange