Node count > 1000 causes failure to schedule

[gisjedi]

Description
When a Scale instance is running with a pre-existing database that includes over 1000 nodes, nothing will schedule if all ready instances are above the initial 1000 records returned. The problem appears to be related to the fact that Scale only requests the first 1000 records from the node table and so it is unable to match the offers to the nodes tracked in memory.

One potential solution is to just update the maximum records returned, the best solution would be to page over all the active nodes.

Reproduction Steps
Steps to reproduce the problem:

1. Create at least 1000 nodes with IPs that aren't present in cluster in the nodes table
2. Launch Scheduler and see offers incoming from new nodes
3. Queue a couple test jobs and observe that they are never scheduled.

[cshamis]

This is insane. 1. Yes. We *only* care about active nodes. 2. Inactive nodes are of historical importance only. 3. We will *never* have 1000 active nodes. 4. Ergo, we don't support more than 1000 active nodes.

…

On Tue, Nov 19, 2019 at 11:44 AM Jonathan Meyer ***@***.***> wrote: *Description* When a Scale instance is running with a pre-existing database that includes over 1000 nodes, nothing will schedule if all ready instances are above the initial 1000 records returned. The problem appears to be related to the fact that Scale only requests the first 1000 records from the node table and so it is unable to match the offers to the nodes tracked in memory. One potential solution is to just update the maximum records returned, the best solution would be to page over all the active nodes. *Reproduction Steps* Steps to reproduce the problem: 1. Go to '...' 2. Click on '....' 3. Scroll down to '....' 4. See error *Expected behavior* A clear and concise description of what you expected to happen. *Screenshots* If applicable, add screenshots to help explain your problem. *Version and Environment Details* - Client OS: [e.g. Windows 10 build 1601] - Browser: [e.g. chrome 76.x, etc.] - Scale API: [e.g. v6.9.0-snapshot+1d15ca7e] - Can be found at /api/v6/version/ - Scale UI: [e.g. v0.4.3] - Can be found at the bottom of the UI interface. Cluster details below are very helpful for problem resolution, but external Scale users aren't expected to provide. - DC/OS: [e.g. 1.11.2 Enterprise, 1.12.0 Community, etc.] - Marathonlb: [e.g. 1.4.2] - Agent OS: [e.g. CentOS 7.4] - Agent Docker: [e.g. 1.17.05] - Agent Infrastructure: [e.g. AWS, GovCloud, On-premise] *Additional context* Add any other context about the problem here. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1825?email_source=notifications&email_token=ADZJETFN62IGDUOKHUY4JZLQUQJU7A5CNFSM4JPGFLA2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H2M3AEA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADZJETCMKE6KP625W4OAU3LQUQJU7ANCNFSM4JPGFLAQ> .