Client config using reserved_ports causes evaluations to fail

[andrewduch]

Nomad version

Nomad v0.5.5

Operating system and Environment details

Servers: Ubuntu 16.04
Client Nodes: Windows 2012 R2
Running in Microsoft Azure
Using raw_exec scheduler

Issue

This looks very similar to issue #1046. When we use reserved_ports, evaluations will not succeed and no reason why is provided. After removing reserved_ports from our client configs, evaluations will run correctly and we can create new allocations.

Let me know if there are any other details I can provide to help here. Thanks!

$ nomad status scarletwitch/ms-cluster2nodev2
ID            = scarletwitch/ms-cluster2nodev2
Name          = scarletwitch-ms-cluster2nodev2
Type          = service
Priority      = 80
Datacenters   = qa
Status        = pending
Periodic      = false
Parameterized = false

Summary
Task Group     Queued  Starting  Running  Failed  Complete  Lost
default-group  1       0         0        0       3         0

Allocations
No allocations placed

$ nomad eval-status 1a7dff0e
ID                 = 1a7dff0e
Status             = failed
Status Description = maximum attempts reached (5)
Type               = service
TriggeredBy        = max-plan-attempts
Previous Eval      = e69b313a-2c5a-f155-52fd-4e30bad2b0b2
Priority           = 80
Placement Failures = false

Reproduction steps

1. Setup Nomad Servers
2. Setup a few Windows 2012 client nodes with config similar to below:

bind_addr   = "0.0.0.0"
data_dir    = "D:\\nomad"
region      = "eastus2"
datacenter  = "dev"
name        = "REDACTED"
log_level   = "INFO"

server {
  enabled = false
}

advertise {
  http = "REDACTED"
  rpc  = "REDACTED"
}

addresses {
  http = "1REDACTED"
}

client {
  enabled = true

  options {
      "driver.raw_exec.enable" = "1"
    }

  meta {

  }

  reserved {
    cpu = "500"
    memory = "512"
    disk = "1024"
    reserved_ports = "0-1024,8500-8600,49152-65535"
  }
}

vault {
  enabled = true
  address = "REDACTED"
}

3. Submit jobs and you will see them go into a pending state.
4. Remove reserved_ports from client config and they will be scheduled.

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

Job file (if appropriate)

[dadgar]

With the reserved block in place can you run:
nomad node-status -json <id> on one of the nodes that you believe should be able to receive a placement.

[andrewduch]

node-status json:
nomad-node-status.json.txt

Even after stopping nomad and clearing the alloc and client/alloc directories, evaluations fail until removing the reserved ports and restarting.

[dadgar]

@andrewduch I just tried on Nomad master on Windows Server 2016 and couldn't reproduce:

PS C:\go\src\github.com\hashicorp\nomad> nomad run .\example.nomad
==> Monitoring evaluation "2ee60758"
    Evaluation triggered by job "example"
    Allocation "b7b1f532" created: node "67612de7", group "cache"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "2ee60758" finished with status "complete"
PS C:\go\src\github.com\hashicorp\nomad> nomad status
ID       Type     Priority  Status
example  service  50        running
PS C:\go\src\github.com\hashicorp\nomad> nomad status e
ID            = example
Name          = example
Type          = service
Priority      = 50
Datacenters   = dc1
Status        = running
Periodic      = false
Parameterized = false

�[1mSummary�[0m�[0m
Task Group  Queued  Starting  Running  Failed  Complete  Lost
cache       0       0         1        0       0         0

�[1mAllocations�[0m�[0m
ID        Eval ID   Node ID   Task Group  Desired  Status   Created At
b7b1f532  2ee60758  67612de7  cache       run      running  04/17/17 23:18:23 GMT
PS C:\go\src\github.com\hashicorp\nomad> nomad alloc-status b7
ID                  = b7b1f532
Eval ID             = 2ee60758
Name                = example.cache[0]
Node ID             = 67612de7
Job ID              = example
Client Status       = running
Client Description  = <none>
Desired Status      = run
Desired Description = <none>
Created At          = 04/17/17 23:18:23 GMT

�[1mTask "redis" is "running"�[0m�[0m
Task Resources
CPU        Memory          Disk     IOPS  Addresses
0/500 MHz  26 MiB/256 MiB  300 MiB  0     db: 10.0.2.15:48480

Recent Events:
Time                   Type        Description
04/17/17 23:18:24 GMT  Started     Task started by client
04/17/17 23:18:23 GMT  Task Setup  Building Task Directory
04/17/17 23:18:23 GMT  Received    Task received by client
PS C:\go\src\github.com\hashicorp\nomad>

I ran Nomad as follows: $ nomad agent -dev -config demo/vagrant/client1.hcl where client1.hcl has this content:

log_level = "DEBUG"
data_dir = "/tmp/client1"
enable_debug = true
name = "client1"
client {
    enabled = true
    servers = ["127.0.0.1:4647"]
    node_class = "foo"
    options {
        "driver.raw_exec.enable" = "1"
    }
    reserved {
       cpu = 500
       reserved_ports = "0-1024,8500-8600,49152-65535"
    }
}

The job I ran was the same as $ nomad init except I changed it to raw_exec and the command to sleep 1000. It was still requesting ports however.

Can you add anymore detail as to what your job is requesting?

[dadgar]

@andrewduch Also just tried on 0.5.5 and same results.

[dadgar]

Closing as this was an older issue that has been fixed.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.