Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add descriptive error message to assert in WorkerState._transition_cancelled_error #6871

Merged
merged 1 commit into from
Aug 11, 2022

Conversation

hendrikmakait
Copy link
Member

@hendrikmakait hendrikmakait commented Aug 11, 2022

This PR adds a descriptive error message to the assert statement in WorkerState._transition_cancelled_error to be able to see the actual ts.previous.

For context: When scaling up tensordot_stress in coiled/benchmarks#229 I ran into the assert statement which returned a not-so-helpful

Aug 11 10:00:05 ip-10-0-3-173 cloud-init[1264]:     assert ts.previous in ("executing", "long-running")
Aug 11 10:00:05 ip-10-0-3-173 cloud-init[1264]: AssertionError
  • Tests added / passed
  • Passes pre-commit run --all-files

@hendrikmakait hendrikmakait self-assigned this Aug 11, 2022
@hendrikmakait hendrikmakait force-pushed the add-descriptive-error-message branch from afdd055 to 7734557 Compare August 11, 2022 10:44
@fjetter
Copy link
Member

fjetter commented Aug 11, 2022

Do you, by chance, have a cluster dump of the cluster where this exception was raised?

@fjetter fjetter merged commit d234569 into dask:main Aug 11, 2022
@hendrikmakait
Copy link
Member Author

@fjetter: I'm trying to recreate this. Unfortunately, I tore down the cluster before stumbling across this in the logs.

@crusaderky
Copy link
Collaborator

crusaderky commented Aug 11, 2022

ts.previous = flight. I can confidently state this without a dump or a reproducer. Discussion in #6877.

gjoseph92 pushed a commit to gjoseph92/distributed that referenced this pull request Oct 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants