Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump kr8s to 0.14.2 #880

Merged
merged 5 commits into from
Apr 19, 2024
Merged

Bump kr8s to 0.14.2 #880

merged 5 commits into from
Apr 19, 2024

Conversation

jacobtomlinson
Copy link
Member

Closes #853

@jacobtomlinson jacobtomlinson changed the title Bump kr8s to 0.14.1 Bump kr8s to 0.14.2 Apr 19, 2024
@jacobtomlinson
Copy link
Member Author

jacobtomlinson commented Apr 19, 2024

I spent some time in #883 trying to figure out what change in kr8s was causing the CI deadlock.

It looks like the problem is because when we run the dask-kubernetes test suite both the controller and client code run within the same process, but on separate event loops. The client caching in kr8s meant that the same API client and other async objects was being shared between these loops, which caused the deadlock. Annoyingly this isn't a real world use case for dask-kubernetes and only happens in the test suite.

There was a fix in kr8s a while back which ensured that the client caching checked the current thread to ensure this doesn't happen, due to an assumption that different event loops must exist in different threads. However that seems to not be true in this case and both loops were reporting the same thread ID.

I tested out a fix that also take the event loop id into account directly from the kr8s PR in #883 and things seem to be passing. So I've made a v0.14.2 release of kr8s and updated this PR to point to the new release which hopefully gets things unblocked.

@jacobtomlinson jacobtomlinson marked this pull request as ready for review April 19, 2024 09:27
@jacobtomlinson jacobtomlinson merged commit 67a45e9 into dask:main Apr 19, 2024
25 checks passed
@jacobtomlinson jacobtomlinson deleted the kr8s-14 branch April 19, 2024 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update dask-kubernetes to a newer kr8s
1 participant