Avoid respawning task trackers constantly when they are idle #59
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously the "idle check" would be run against all task trackers regardless of whether they have any jobs assigned to them or not. The main MesosScheduler is responsible for cleaning up task trackers once jobs have *finished* so this change stops us performing idle checks on trackers that have no jobs. This change fixes the observed behaviour of task trackers being killed and respawning continuously when they're waiting for jobs (e.g with the min map/reduce slot config option, or the fixed resource policy).
tarnfeld
added a commit
that referenced
this pull request
May 20, 2015
Avoid respawning task trackers constantly when they are idle
|
Note that this was reverted from master |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously the "idle check" would be run against all task trackers regardless of whether they have any jobs assigned to them or not. The main MesosScheduler is responsible for cleaning up task trackers once jobs have finished so this change stops us performing idle checks on trackers that have no jobs.
Destruction of those task trackers will be handled elsewhere.
This change fixes the observed behaviour of task trackers being killed and respawning continuously when they're waiting for jobs to be scheduled on them (e.g with the min map/reduce slot config option, or the fixed resource policy). Essentially this fixed a case when trackers were killed due to being idle, when they were supposed to be sitting idle.