Add support for disk and network_bandwidth resources to run in Mesos at Criteo #87
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Backport in Spark 2.3 Before this change, there was no way to allocate a given amount of disk when using Mesos scheduler. It's good enough when using default isolation options but not when enabling the XFS isolator with hard limit in order to properly isolate all containers. In that case, the executor is killed by Mesos during the download of the Spark executor archive. Therefore, this change introduces a configuration flag, specific to Mesos, to declare the amount of disk required by the executors and therefore prevent Mesos from killing the container because the XFS hard limit has been exceeded.
clems4ever
changed the title
At criteo, containers are network isolated and if no network bandwidth resource is allocated, Mesos will reserve a default amount. This patch provide a way for user to specify the amount of network bandwidth required by the executors and therefore used by the tasks. This fix is Criteo specific as network_bandwidth is a custom resource we introduced in our clusters and the code of the isolator is not public (yet).
clems4ever
force-pushed
the
mesos-disk-criteo-2.3
branch
from
9bf9816 to
44981a7
Compare
Author
|
I ran the Mesos related tests on my machine to confirm my fix is ok since the CI build is failing for other reasons. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is made of 2 patches:
Fix 1. is a backport in Spark 2.3 from a PR in 3.0 (apache#23758)
Fix 2. is completely Criteo specific since it relies on a resource that we introduced ourselves.