-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Single-mode exceptions in isImpassable / Limit launched workers to half of remaining capacity #848
Conversation
Allows config.maxWorkers() and MAX_WORKERS_PER_CATEGORY to be more freely set on staging.
I have added another unrelated change to this PR to apply different limits when starting large numbers of spot instances. This gives us a version of R5 with both patches applied for immediate testing. |
Trying out the worker limiting on staging: on running a first analysis I see in the logs:
Then, on running a second analysis:
Note that 67 is more than half of the 123 capacity that should have been remaining after the first run. This is probably because there's a time lag in the backend becoming aware of the newly started workers - not all of them had reported in yet when I ran the second job, so |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the revised worker startup logic and node tag checks.
src/main/java/com/conveyal/analysis/components/broker/Broker.java
Outdated
Show resolved
Hide resolved
Co-authored-by: Anson Stewart <astewart@conveyal.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good
See discussion in comments on #841 and in code comments. Even when access=no|private, there are frequently exceptions for single modes. We want to default to the preexisting behavior of allowing passage (and rely only on edge traversal permissions) whenever we see exception tags that aren't clearly blocking access.