Better preferential scheduling

[sjeropkipruto]

Status: Work in progress 🚧

Description

Mesos hosts aren't nearly as well utilized as they could be. The high mem ones fill up on cpu before we can make full use of the extra memory and vice versa for the high cpu ones. Similar for spot, not all things that can run on spot actually are.

We want to improve our scheduling to really improve our utilization and make scheduling faster & easier because there are more available large spots of resources.

Testing

• unit testing
• deploying to staging

Todos

• Implement preference by allowed attributes. If a deployable specifies an allowed attribute, give preference to that type of machine. For example, if it can run on spot, prefer spot. If it is allowed to run on spark hosts, prefer those.
• Implement preference by resource preference. Identify mesos hosts as normal or having a particular resource preference (e.g. a high memory host). Do the same for tasks based on their measured used resources. Matches between tasks and hosts should be preferred (e.g. high memory task on a high memory host).
• Write unit tests
• Deploy and test in staging
• Deploy to QA
• Deploy to PROD

[ssalinas]

Overall concept looks good 👍 , let's get some unit tests in to make sure it behaves as expected then give it a go in staging

[ssalinas]

🚢