Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement --min-chunksize #154

Open
gdevenyi opened this issue Apr 23, 2018 · 2 comments
Open

Implement --min-chunksize #154

gdevenyi opened this issue Apr 23, 2018 · 2 comments
Milestone

Comments

@gdevenyi
Copy link
Member

gdevenyi commented Apr 23, 2018

If I have something like a joblist of 81 commands and my chunksize is 20, I get this unfortunate result of jobs with 40,40,1. This is bad for the cluster.

Should be able to define a tolerance of number of commands (percent?) for overpacking the last job. Requires a little bit of extra logic to handle it but probably worth the effort.

@pipitone
Copy link
Collaborator

How about a --min-chunksize option defaulting to 1. If the last chunk size is < the min-chunk size, you add those tasks to the previous chunk.

@gdevenyi
Copy link
Member Author

Ah, that's a much clearer naming than tolerance 👍

@gdevenyi gdevenyi changed the title Implement chunksize tolerance Implement --min-chunksize May 2, 2018
@gdevenyi gdevenyi added this to the v2.2 milestone Jun 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants