Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resource monitor does not work with coffea casa #3810

Open
btovar opened this issue May 2, 2024 · 8 comments
Open

resource monitor does not work with coffea casa #3810

btovar opened this issue May 2, 2024 · 8 comments
Labels
bug For modifications that fix a flaw in the code. Resource Monitor

Comments

@btovar
Copy link
Member

btovar commented May 2, 2024

Tasks fail with:

The futex facility returned an unexpected error code.
@btovar btovar added bug For modifications that fix a flaw in the code. Resource Monitor labels May 2, 2024
@btovar
Copy link
Member Author

btovar commented May 2, 2024

Sometimes it works. It looks like an ugly race condition, not sure from where.

@dthain
Copy link
Member

dthain commented May 2, 2024

Wow, that's a scary one. Is the RM even interposing on sync-related things?

@btovar
Copy link
Member Author

btovar commented May 2, 2024

It is not as far as I can tell. It might be that it adds enough overhead to trigger something. I'm not sure if it is because we are running inside a container, or it is something that python is doing...

@btovar
Copy link
Member Author

btovar commented May 3, 2024

The resource_monitor does not work, but the monitored decorator inside python works!

@btovar
Copy link
Member Author

btovar commented May 3, 2024

Ah, spoke too soon. It doesn't work for all functions, probably the same issue.

@btovar
Copy link
Member Author

btovar commented May 6, 2024

To add to the mystery. If the task is within its resources, then it runs correctly to completion and its resources are correctly measured.

@dthain
Copy link
Member

dthain commented Jun 12, 2024

Maybe this is some interaction with container technologies?

@btovar
Copy link
Member Author

btovar commented Jun 12, 2024

I think so. "Unfortunately" our tests with github actions that run inside containers seem to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug For modifications that fix a flaw in the code. Resource Monitor
Projects
Status: No status
Development

No branches or pull requests

2 participants