Probabilistic priority queueing

[lenhattan86]

Description:

The change enables soft priority queueing for aurora-scheduler.
If pending Task A has higher priority than other pending tasks, it has more chance to be scheduled than other pending tasks.
However, it does not mean that low priority tasks have to wait until all higher priority tasks scheduled. So, we do not starve any pending tasks.
The chance of scheduling is calculated using an exponential function p^exp where p is the priority and exp is the exponent.
We control exp using probabilistic_priority_assigner_exponent.

If exp is 0, there is no priority queueing.
if exp is high, there is less chance that the lower priority tasks can be scheduled.

Testing Done:

• unit tests
• local cluster tests.

[zorro786]

First quick pass

[zorro786]

Second pass

[zorro786]

@lenhattan86 There are several commits in this PR that are not related, it must be from a stale branch merge. Once those are fixed, PR LGTM.

[lenhattan86]

@lenhattan86 There are several commits in this PR that are not related, it must be from a stale branch merge. Once those are fixed, PR LGTM.

right, they are merged from master to fix other issues. However, none of them cause conflicts if you look at the list of changed files. https://github.com/aurora-scheduler/scheduler/pull/322/files

[zorro786]

@lenhattan86 There are several commits in this PR that are not related, it must be from a stale branch merge. Once those are fixed, PR LGTM.

right, they are merged from master to fix other issues. However, none of them cause conflicts if you look at the list of changed files. https://github.com/aurora-scheduler/scheduler/pull/322/files

I am not worried about conflicts, but for commit history and number of commits/messages. Once squashed and merged those older commit messages will also be part of single commit.

[lenhattan86]

@lenhattan86 There are several commits in this PR that are not related, it must be from a stale branch merge. Once those are fixed, PR LGTM.

right, they are merged from master to fix other issues. However, none of them cause conflicts if you look at the list of changed files. https://github.com/aurora-scheduler/scheduler/pull/322/files

I am not worried about conflicts, but for commit history and number of commits/messages. Once squashed and merged those older commit messages will also be part of single commit.

sure will clean them up when merging

[zorro786]

LGTM

[ridv]

Minor nits: LGTM