dns_search_domain not applied when using bridge network mode #11239

jhaezebr · 2021-09-29T17:01:00Z

Nomad version

Nomad v1.1.5 (117a23d)

Operating system and Environment details

Vagrant with ubuntu/focal64
CNI bridge plugin v1.0.1
Consul v1.9.4, Revision 10bb6cb3b

Issue

When running a job using the bridge network mode, the dns_search_domains parameter is not applied.

Reproduction steps

Run nomad and consul as root, both in dev mode
Use bind to redirect DNS to consul

Expected Result

$ nomad exec -job nonbridged /bin/sh -c 'cat /etc/resolv.conf && echo "--------" && nslookup nomad'
search service.consul
nameserver 10.0.0.2
--------
Server:    10.0.0.2
Address 1: 10.0.0.2

Name:      nomad
Address 1: 10.0.2.15

Actual Result

$ nomad exec -job bridged /bin/sh -c 'cat /etc/resolv.conf && echo "--------" && nslookup nomad'
search example.com
nameserver 10.0.0.2
--------
Server:    10.0.0.2
Address 1: 10.0.0.2

nslookup: can't resolve 'nomad'

Job file (if appropriate)

Bridged job:

job "bridged" {
  datacenters = ["dc1"]
  group "bridged" {
    network {
      mode = "bridge"
    }
    task "bridged" {
      driver = "docker"
      config {
        image = "busybox:1.28"
        dns_search_domains = ["service.consul"]
        command = "sleep"
        args = ["3600"]
      }
    }
  }
}

Non bridged job:

job "nonbridged" {
  datacenters = ["dc1"]
  group "nonbridged" {
    task "nonbridged" {
      driver = "docker"
      config {
        image = "busybox:1.28"
        dns_search_domains = ["service.consul"]
        command = "sleep"
        args = ["3600"]
      }
    }
  }
}

Nomad logs

Click to expand


==> Loaded configuration from nomad.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:
   Advertise Addrs: HTTP: 10.0.2.15:4646; RPC: 10.0.2.15:4647; Serf: 10.0.2.15:4648
        Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
            Client: true
         Log Level: DEBUG
            Region: global (DC: dc1)
            Server: true
           Version: 1.1.5

==> Nomad agent started! Log data will stream in below:
2021-09-29T16:31:45.361Z [WARN]  agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/var/lib/nomad/plugins
2021-09-29T16:31:45.362Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/plugins
2021-09-29T16:31:45.362Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/plugins
2021-09-29T16:31:45.362Z [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
2021-09-29T16:31:45.362Z [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
2021-09-29T16:31:45.362Z [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
2021-09-29T16:31:45.362Z [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
2021-09-29T16:31:45.362Z [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
2021-09-29T16:31:45.362Z [INFO]  agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
2021-09-29T16:31:45.364Z [INFO]  nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:10.0.2.15:4647 Address:10.0.2.15:4647}]"
2021-09-29T16:31:45.364Z [INFO]  nomad: serf: EventMemberJoin: server.global 10.0.2.15
2021-09-29T16:31:45.364Z [INFO]  nomad: starting scheduling worker(s): num_workers=2 schedulers=[service, batch, system, _core]
2021-09-29T16:31:45.364Z [INFO]  client: using state directory: state_dir=/var/lib/nomad/client
2021-09-29T16:31:45.364Z [INFO]  client: using alloc directory: alloc_dir=/var/lib/nomad/alloc
2021-09-29T16:31:45.364Z [INFO]  nomad.raft: entering follower state: follower="Node at 10.0.2.15:4647 [Follower]" leader=
2021-09-29T16:31:45.365Z [INFO]  nomad: adding server: server="server.global (Addr: 10.0.2.15:4647) (DC: dc1)"
2021-09-29T16:31:45.376Z [INFO]  client.cpuset_manager: initialized cpuset cgroup manager: parent=/nomad cpuset=0-1
2021-09-29T16:31:45.377Z [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=[arch, bridge, cgroup, cni, consul, cpu, host, memory, network, nomad, signal, storage, vault, env_aws, env_gce, env_azure]
2021-09-29T16:31:45.377Z [INFO]  client.fingerprint_mgr.cgroup: cgroups are available
2021-09-29T16:31:45.377Z [DEBUG] client.fingerprint_mgr: CNI config dir is not set or does not exist, skipping: cni_config_dir=/opt/cni/config
2021-09-29T16:31:45.377Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
2021-09-29T16:31:45.382Z [INFO]  client.fingerprint_mgr.consul: consul agent is available
2021-09-29T16:31:45.383Z [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=1895
2021-09-29T16:31:45.383Z [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=2
2021-09-29T16:31:45.383Z [DEBUG] client.fingerprint_mgr.cpu: detected reservable cores: cpuset=[0, 1]
2021-09-29T16:31:45.384Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
2021-09-29T16:31:45.386Z [DEBUG] consul.sync: sync complete: registered_services=3 deregistered_services=0 registered_checks=3 deregistered_checks=0
2021-09-29T16:31:45.387Z [DEBUG] client.fingerprint_mgr.network: link speed detected: interface=enp0s8 mbits=1000
2021-09-29T16:31:45.387Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s8 IP=10.0.0.2
2021-09-29T16:31:45.389Z [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=lo
2021-09-29T16:31:45.389Z [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/lo/speed
2021-09-29T16:31:45.389Z [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
2021-09-29T16:31:45.392Z [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=docker0
2021-09-29T16:31:45.392Z [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/docker0/speed
2021-09-29T16:31:45.392Z [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
2021-09-29T16:31:45.393Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
2021-09-29T16:31:47.377Z [WARN]  nomad.raft: heartbeat timeout reached, starting election: last-leader=
2021-09-29T16:31:47.377Z [INFO]  nomad.raft: entering candidate state: node="Node at 10.0.2.15:4647 [Candidate]" term=2
2021-09-29T16:31:47.377Z [DEBUG] nomad.raft: votes: needed=1
2021-09-29T16:31:47.377Z [DEBUG] nomad.raft: vote granted: from=10.0.2.15:4647 term=2 tally=1
2021-09-29T16:31:47.377Z [INFO]  nomad.raft: election won: tally=1
2021-09-29T16:31:47.377Z [INFO]  nomad.raft: entering leader state: leader="Node at 10.0.2.15:4647 [Leader]"
2021-09-29T16:31:47.378Z [INFO]  nomad: cluster leadership acquired
2021-09-29T16:31:47.382Z [INFO]  nomad.core: established cluster id: cluster_id=2d76fc62-7b7b-b0f8-918e-677286811f96 create_time=1632933107382405371
2021-09-29T16:31:50.492Z [DEBUG] client.fingerprint_mgr.env_gce: could not read value for attribute: attribute=machine-type error="Get "http://169.254.169.254/computeMetadata/v1/instance/machine-type": dial tcp 169.254.169.254:80: i/o timeout (Client.Timeout exceeded while awaiting headers)"
2021-09-29T16:31:50.492Z [DEBUG] client.fingerprint_mgr.env_gce: error querying GCE Metadata URL, skipping
2021-09-29T16:31:51.544Z [DEBUG] client.fingerprint_mgr.env_azure: could not read value for attribute: attribute=compute/azEnvironment error="Get "http://169.254.169.254/metadata/instance/compute/azEnvironment?api-version=2019-06-04&format=text": dial tcp 169.254.169.254:80: connect: no route to host"
2021-09-29T16:31:51.544Z [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs=[arch, bridge, cgroup, consul, cpu, host, network, nomad, signal, storage]
2021-09-29T16:31:51.544Z [INFO]  client.plugin: starting plugin manager: plugin-type=csi
2021-09-29T16:31:51.545Z [INFO]  client.plugin: starting plugin manager: plugin-type=driver
2021-09-29T16:31:51.545Z [INFO]  client.plugin: starting plugin manager: plugin-type=device
2021-09-29T16:31:51.548Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=java health=undetected description=
2021-09-29T16:31:51.549Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=raw_exec health=healthy description=Healthy
2021-09-29T16:31:51.550Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=exec health=healthy description=Healthy
2021-09-29T16:31:51.550Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=qemu health=undetected description=
2021-09-29T16:31:51.550Z [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=driver
2021-09-29T16:31:51.550Z [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=device
2021-09-29T16:31:51.550Z [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=device
2021-09-29T16:31:51.553Z [DEBUG] client.server_mgr: new server list: new_servers=[0.0.0.0:4647, 10.0.2.15:4647] old_servers=[]
2021-09-29T16:31:51.593Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=healthy description=Healthy
2021-09-29T16:31:51.593Z [DEBUG] client.driver_mgr: detected drivers: drivers="map[healthy:[raw_exec exec docker] undetected:[qemu java]]"
2021-09-29T16:31:51.594Z [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver
2021-09-29T16:31:51.594Z [INFO]  client: started client: node_id=23b96b94-8b8e-92fa-4309-06da749bc6d6
2021-09-29T16:31:51.596Z [INFO]  client: node registration complete
2021-09-29T16:31:51.596Z [DEBUG] client: updated allocations: index=1 total=0 pulled=0 filtered=0
2021-09-29T16:31:51.596Z [DEBUG] client: state updated: node_status=ready
2021-09-29T16:31:51.596Z [DEBUG] client.server_mgr: new server list: new_servers=[10.0.2.15:4647] old_servers=[10.0.2.15:4647, 0.0.0.0:4647]
2021-09-29T16:31:51.596Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=0
2021-09-29T16:31:51.597Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=0 errors=0
2021-09-29T16:31:51.611Z [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=1 deregistered_checks=0
2021-09-29T16:31:51.899Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404
2021-09-29T16:31:51.899Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.485728ms
2021-09-29T16:31:52.597Z [DEBUG] client: state changed, updating node and re-registering
2021-09-29T16:31:52.602Z [INFO]  client: node registration complete
2021-09-29T16:31:54.917Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404
2021-09-29T16:31:54.917Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=933.93µs
2021-09-29T16:31:55.267Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44196
2021-09-29T16:31:55.834Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=636.632µs
2021-09-29T16:31:57.932Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404
2021-09-29T16:31:57.933Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.123522ms
2021-09-29T16:31:58.127Z [DEBUG] http: request complete: method=PUT path=/v1/jobs duration=1.811224ms
2021-09-29T16:31:58.127Z [DEBUG] worker: dequeued evaluation: eval_id=87540f34-e641-853f-a1ca-1b3c2b31ae7e
2021-09-29T16:31:58.128Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=87540f34-e641-853f-a1ca-1b3c2b31ae7e job_id=nonbridged namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 0)

Created Deployment: "d91b08c6-71e4-c5ab-5a44-e4d346f6fb80"

Desired Changes for "nonbridged": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)"

2021-09-29T16:31:58.130Z [DEBUG] worker: submitted plan for evaluation: eval_id=87540f34-e641-853f-a1ca-1b3c2b31ae7e

2021-09-29T16:31:58.131Z [DEBUG] http: request complete: method=GET path=/v1/evaluation/87540f34-e641-853f-a1ca-1b3c2b31ae7e duration=622.174µs

2021-09-29T16:31:58.131Z [DEBUG] client: updated allocations: index=11 total=1 pulled=1 filtered=0

2021-09-29T16:31:58.132Z [DEBUG] worker.service_sched: setting eval status: eval_id=87540f34-e641-853f-a1ca-1b3c2b31ae7e job_id=nonbridged namespace=default status=complete

2021-09-29T16:31:58.133Z [DEBUG] client: allocation updates: added=1 removed=0 updated=0 ignored=0

2021-09-29T16:31:58.134Z [DEBUG] http: request complete: method=GET path=/v1/evaluation/87540f34-e641-853f-a1ca-1b3c2b31ae7e/allocations duration=358.177µs

2021-09-29T16:31:58.134Z [DEBUG] worker: updated evaluation: eval="<Eval "87540f34-e641-853f-a1ca-1b3c2b31ae7e" JobID: "nonbridged" Namespace: "default">"

2021-09-29T16:31:58.136Z [DEBUG] worker: ack evaluation: eval_id=87540f34-e641-853f-a1ca-1b3c2b31ae7e

2021-09-29T16:31:58.137Z [DEBUG] client: allocation updates applied: added=1 removed=0 updated=0 ignored=0 errors=0

2021-09-29T16:31:58.138Z [DEBUG] client.alloc_runner.task_runner: lifecycle start condition has been met, proceeding: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged

2021-09-29T16:31:58.142Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged path=/usr/bin/nomad args=[/usr/bin/nomad, logmon]

2021-09-29T16:31:58.143Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged path=/usr/bin/nomad pid=18366

2021-09-29T16:31:58.143Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged path=/usr/bin/nomad

2021-09-29T16:31:58.151Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged address=/tmp/plugin034140996 network=unix @module=logmon timestamp=2021-09-29T16:31:58.151Z

2021-09-29T16:31:58.152Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged version=2

2021-09-29T16:31:58.162Z [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged @module=logmon path=/var/lib/nomad/alloc/5c9fad52-2e79-e3b6-74b1-1414277695ce/alloc/logs/.nonbridged.stdout.fifo timestamp=2021-09-29T16:31:58.162Z

2021-09-29T16:31:58.163Z [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged @module=logmon path=/var/lib/nomad/alloc/5c9fad52-2e79-e3b6-74b1-1414277695ce/alloc/logs/.nonbridged.stderr.fifo timestamp=2021-09-29T16:31:58.163Z

2021-09-29T16:31:58.167Z [DEBUG] client.driver_mgr.docker: image reference count incremented: driver=docker image_name=busybox:1.28 image_id=sha256:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a references=1

2021-09-29T16:31:58.167Z [DEBUG] client.driver_mgr.docker: configured resources: driver=docker task_name=nonbridged memory=314572800 memory_reservation=0 cpu_shares=100 cpu_quota=0 cpu_period=0

2021-09-29T16:31:58.167Z [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=nonbridged binds="[]string{"/var/lib/nomad/alloc/5c9fad52-2e79-e3b6-74b1-1414277695ce/alloc:/alloc", "/var/lib/nomad/alloc/5c9fad52-2e79-e3b6-74b1-1414277695ce/nonbridged/local:/local", "/var/lib/nomad/alloc/5c9fad52-2e79-e3b6-74b1-1414277695ce/nonbridged/secrets:/secrets"}"

2021-09-29T16:31:58.168Z [DEBUG] client.driver_mgr.docker: networking mode not specified; using default: driver=docker task_name=nonbridged

2021-09-29T16:31:58.168Z [DEBUG] client.driver_mgr.docker: setting container startup command: driver=docker task_name=nonbridged command="sleep 3600"

2021-09-29T16:31:58.169Z [DEBUG] client.driver_mgr.docker: applied labels on the container: driver=docker task_name=nonbridged labels=map[com.hashicorp.nomad.alloc_id:5c9fad52-2e79-e3b6-74b1-1414277695ce]

2021-09-29T16:31:58.169Z [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=nonbridged container_name=nonbridged-5c9fad52-2e79-e3b6-74b1-1414277695ce

2021-09-29T16:31:58.187Z [INFO]  client.driver_mgr.docker: created container: driver=docker container_id=32926edfa3458cb5e2c5d36e74fe3d034a00c6abc658754879b4c905696f1a81

2021-09-29T16:31:58.246Z [DEBUG] client: updated allocations: index=13 total=1 pulled=0 filtered=1

2021-09-29T16:31:58.247Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1

2021-09-29T16:31:58.247Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0

2021-09-29T16:31:58.364Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=161.118µs

2021-09-29T16:31:58.495Z [INFO]  client.driver_mgr.docker: started container: driver=docker container_id=32926edfa3458cb5e2c5d36e74fe3d034a00c6abc658754879b4c905696f1a81

2021-09-29T16:31:58.500Z [DEBUG] client.driver_mgr.docker.docker_logger: starting plugin: driver=docker path=/usr/bin/nomad args=[/usr/bin/nomad, docker_logger]

2021-09-29T16:31:58.501Z [DEBUG] client.driver_mgr.docker.docker_logger: plugin started: driver=docker path=/usr/bin/nomad pid=18469

2021-09-29T16:31:58.501Z [DEBUG] client.driver_mgr.docker.docker_logger: waiting for RPC address: driver=docker path=/usr/bin/nomad

2021-09-29T16:31:58.509Z [DEBUG] client.driver_mgr.docker.docker_logger: using plugin: driver=docker version=2

2021-09-29T16:31:58.509Z [DEBUG] client.driver_mgr.docker.docker_logger.nomad: plugin address: driver=docker network=unix @module=docker_logger address=/tmp/plugin008383632 timestamp=2021-09-29T16:31:58.509Z

2021-09-29T16:31:58.511Z [DEBUG] client.driver_mgr.docker.docker_logger.nomad: using client connection initialized from environment: driver=docker @module=docker_logger timestamp=2021-09-29T16:31:58.511Z

2021-09-29T16:31:58.702Z [DEBUG] client: updated allocations: index=14 total=1 pulled=0 filtered=1

2021-09-29T16:31:58.703Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1

2021-09-29T16:31:58.703Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0

2021-09-29T16:31:59.139Z [DEBUG] http: request complete: method=GET path=/v1/evaluation/87540f34-e641-853f-a1ca-1b3c2b31ae7e duration=635.727µs

2021-09-29T16:31:59.145Z [DEBUG] http: request complete: method=GET path=/v1/evaluation/87540f34-e641-853f-a1ca-1b3c2b31ae7e/allocations duration=857.108µs

2021-09-29T16:31:59.155Z [DEBUG] http: request complete: method=GET path=/v1/deployment/d91b08c6-71e4-c5ab-5a44-e4d346f6fb80?stale=&wait=2000ms duration=745.065µs

2021-09-29T16:32:00.954Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:00.955Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.849139ms

2021-09-29T16:32:01.286Z [DEBUG] http: request complete: method=GET path=/v1/deployment/d91b08c6-71e4-c5ab-5a44-e4d346f6fb80?index=11&stale=&wait=2000ms duration=2.125555392s

2021-09-29T16:32:03.327Z [DEBUG] http: request complete: method=GET path=/v1/deployment/d91b08c6-71e4-c5ab-5a44-e4d346f6fb80?index=11&stale=&wait=2000ms duration=2.036449502s

2021-09-29T16:32:03.978Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:03.979Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.374557ms

2021-09-29T16:32:05.271Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44222

2021-09-29T16:32:05.440Z [DEBUG] http: request complete: method=GET path=/v1/deployment/d91b08c6-71e4-c5ab-5a44-e4d346f6fb80?index=11&stale=&wait=2000ms duration=2.108302961s

2021-09-29T16:32:05.842Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=456.81µs

2021-09-29T16:32:06.999Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:07.000Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.214761ms

2021-09-29T16:32:07.476Z [DEBUG] http: request complete: method=GET path=/v1/deployment/d91b08c6-71e4-c5ab-5a44-e4d346f6fb80?index=11&stale=&wait=2000ms duration=2.028747508s

2021-09-29T16:32:08.375Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=701.341µs

2021-09-29T16:32:08.755Z [DEBUG] http: request complete: method=GET path=/v1/deployment/d91b08c6-71e4-c5ab-5a44-e4d346f6fb80?index=11&stale=&wait=2000ms duration=1.274224407s

2021-09-29T16:32:08.757Z [DEBUG] client: updated allocations: index=15 total=1 pulled=0 filtered=1

2021-09-29T16:32:08.760Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1

2021-09-29T16:32:08.761Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0

2021-09-29T16:32:10.007Z [DEBUG] worker: dequeued evaluation: eval_id=aa31a9e1-9829-4ea7-3785-abde1ebce618

2021-09-29T16:32:10.007Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=aa31a9e1-9829-4ea7-3785-abde1ebce618 job_id=nonbridged namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0)

Deployment Update for ID "d91b08c6-71e4-c5ab-5a44-e4d346f6fb80": Status "successful"; Description "Deployment completed successfully"

Desired Changes for "nonbridged": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)"

2021-09-29T16:32:10.009Z [DEBUG] http: request complete: method=GET path=/v1/deployment/d91b08c6-71e4-c5ab-5a44-e4d346f6fb80?index=15&stale=&wait=2000ms duration=1.244021966s

2021-09-29T16:32:10.010Z [DEBUG] worker: submitted plan for evaluation: eval_id=aa31a9e1-9829-4ea7-3785-abde1ebce618

2021-09-29T16:32:10.011Z [DEBUG] worker.service_sched: setting eval status: eval_id=aa31a9e1-9829-4ea7-3785-abde1ebce618 job_id=nonbridged namespace=default status=complete

2021-09-29T16:32:10.012Z [DEBUG] worker: updated evaluation: eval="<Eval "aa31a9e1-9829-4ea7-3785-abde1ebce618" JobID: "nonbridged" Namespace: "default">"

2021-09-29T16:32:10.013Z [DEBUG] worker: ack evaluation: eval_id=aa31a9e1-9829-4ea7-3785-abde1ebce618

2021-09-29T16:32:10.015Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:10.015Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=738.132µs

2021-09-29T16:32:13.030Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:13.032Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.536172ms

2021-09-29T16:32:15.280Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44236

2021-09-29T16:32:15.847Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=462.692µs

2021-09-29T16:32:16.051Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:16.052Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.045832ms

2021-09-29T16:32:17.019Z [DEBUG] http: request complete: method=PUT path=/v1/jobs duration=699.024µs

2021-09-29T16:32:17.019Z [DEBUG] worker: dequeued evaluation: eval_id=863b1ab6-8db8-c144-7870-05eec7f625f7

2021-09-29T16:32:17.020Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=863b1ab6-8db8-c144-7870-05eec7f625f7 job_id=bridged namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 0)

Created Deployment: "e83aaa10-073c-7aa0-f858-1a03e44362aa"

Desired Changes for "bridged": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)"

2021-09-29T16:32:17.021Z [DEBUG] http: request complete: method=GET path=/v1/evaluation/863b1ab6-8db8-c144-7870-05eec7f625f7 duration=245.364µs

2021-09-29T16:32:17.023Z [DEBUG] worker: submitted plan for evaluation: eval_id=863b1ab6-8db8-c144-7870-05eec7f625f7

2021-09-29T16:32:17.023Z [DEBUG] worker.service_sched: setting eval status: eval_id=863b1ab6-8db8-c144-7870-05eec7f625f7 job_id=bridged namespace=default status=complete

2021-09-29T16:32:17.023Z [DEBUG] client: updated allocations: index=20 total=2 pulled=1 filtered=1

2021-09-29T16:32:17.023Z [DEBUG] http: request complete: method=GET path=/v1/evaluation/863b1ab6-8db8-c144-7870-05eec7f625f7/allocations duration=208.351µs

2021-09-29T16:32:17.023Z [DEBUG] worker: updated evaluation: eval="<Eval "863b1ab6-8db8-c144-7870-05eec7f625f7" JobID: "bridged" Namespace: "default">"

2021-09-29T16:32:17.024Z [DEBUG] client: allocation updates: added=1 removed=0 updated=0 ignored=1

2021-09-29T16:32:17.024Z [DEBUG] worker: ack evaluation: eval_id=863b1ab6-8db8-c144-7870-05eec7f625f7

2021-09-29T16:32:17.026Z [DEBUG] client: allocation updates applied: added=1 removed=0 updated=0 ignored=1 errors=0

2021-09-29T16:32:18.029Z [DEBUG] http: request complete: method=GET path=/v1/evaluation/863b1ab6-8db8-c144-7870-05eec7f625f7 duration=800.873µs

2021-09-29T16:32:18.035Z [DEBUG] http: request complete: method=GET path=/v1/evaluation/863b1ab6-8db8-c144-7870-05eec7f625f7/allocations duration=1.14181ms

2021-09-29T16:32:18.038Z [DEBUG] http: request complete: method=GET path=/v1/deployment/e83aaa10-073c-7aa0-f858-1a03e44362aa?stale=&wait=2000ms duration=684.143µs

2021-09-29T16:32:18.382Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=587.563µs

2021-09-29T16:32:19.074Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:19.075Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=2.278289ms

2021-09-29T16:32:19.986Z [DEBUG] client.driver_mgr.docker: docker pull succeeded: driver=docker image_ref=gcr.io/google_containers/pause-amd64:3.1

2021-09-29T16:32:19.987Z [DEBUG] client.driver_mgr.docker: image reference count incremented: driver=docker image_name=gcr.io/google_containers/pause-amd64:3.1 image_id=sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e references=1

2021-09-29T16:32:20.068Z [DEBUG] http: request complete: method=GET path=/v1/deployment/e83aaa10-073c-7aa0-f858-1a03e44362aa?index=20&stale=&wait=2000ms duration=2.013744545s

2021-09-29T16:32:20.307Z [DEBUG] client.alloc_runner.runner_hook: received result from CNI: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 result={"Interfaces":{"eth0":{"IPConfigs":[{"IP":"172.26.64.2","Gateway":"172.26.64.1"}],"Mac":"96:1f:80:ef:dc:8b","Sandbox":"/var/run/docker/netns/1c13a1ef381e"},"nomad":{"IPConfigs":null,"Mac":"ea:34:78:36:6f:fd","Sandbox":""},"veth1cbbff84":{"IPConfigs":null,"Mac":"46:d3:06:86:d9:90","Sandbox":""}},"DNS":[{}],"Routes":[{"dst":"0.0.0.0/0"}]}

2021-09-29T16:32:20.308Z [DEBUG] client.alloc_runner.task_runner: lifecycle start condition has been met, proceeding: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged

2021-09-29T16:32:20.311Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged path=/usr/bin/nomad args=[/usr/bin/nomad, logmon]

2021-09-29T16:32:20.311Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged path=/usr/bin/nomad pid=18636

2021-09-29T16:32:20.312Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged path=/usr/bin/nomad

2021-09-29T16:32:20.319Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged version=2

2021-09-29T16:32:20.319Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged @module=logmon address=/tmp/plugin642957040 network=unix timestamp=2021-09-29T16:32:20.319Z

2021-09-29T16:32:20.322Z [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged @module=logmon path=/var/lib/nomad/alloc/1a3e8ff1-ca08-633e-42dd-222b44c60734/alloc/logs/.bridged.stdout.fifo timestamp=2021-09-29T16:32:20.321Z

2021-09-29T16:32:20.322Z [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged path=/var/lib/nomad/alloc/1a3e8ff1-ca08-633e-42dd-222b44c60734/alloc/logs/.bridged.stderr.fifo @module=logmon timestamp=2021-09-29T16:32:20.322Z

2021-09-29T16:32:20.324Z [DEBUG] client.driver_mgr.docker: image reference count incremented: driver=docker image_name=busybox:1.28 image_id=sha256:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a references=2

2021-09-29T16:32:20.324Z [DEBUG] client.driver_mgr.docker: configured resources: driver=docker task_name=bridged memory=314572800 memory_reservation=0 cpu_shares=100 cpu_quota=0 cpu_period=0

2021-09-29T16:32:20.325Z [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=bridged binds="[]string{"/var/lib/nomad/alloc/1a3e8ff1-ca08-633e-42dd-222b44c60734/alloc:/alloc", "/var/lib/nomad/alloc/1a3e8ff1-ca08-633e-42dd-222b44c60734/bridged/local:/local", "/var/lib/nomad/alloc/1a3e8ff1-ca08-633e-42dd-222b44c60734/bridged/secrets:/secrets"}"

2021-09-29T16:32:20.325Z [DEBUG] client.driver_mgr.docker: configuring network mode for task group: driver=docker task_name=bridged network_mode=container:8704d6a8683c440a5448df986941fafc79283f3faca541dcb26d841f7e98585d

2021-09-29T16:32:20.325Z [DEBUG] client.driver_mgr.docker: setting container startup command: driver=docker task_name=bridged command="sleep 3600"

2021-09-29T16:32:20.325Z [DEBUG] client.driver_mgr.docker: applied labels on the container: driver=docker task_name=bridged labels=map[com.hashicorp.nomad.alloc_id:1a3e8ff1-ca08-633e-42dd-222b44c60734]

2021-09-29T16:32:20.325Z [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=bridged container_name=bridged-1a3e8ff1-ca08-633e-42dd-222b44c60734

2021-09-29T16:32:20.353Z [INFO]  client.driver_mgr.docker: created container: driver=docker container_id=4862c4921af65432be66510be472808508ecdf23bbf52205d5a2e38702e91035

2021-09-29T16:32:20.408Z [DEBUG] client: updated allocations: index=22 total=2 pulled=0 filtered=2

2021-09-29T16:32:20.409Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=2

2021-09-29T16:32:20.409Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=2 errors=0

2021-09-29T16:32:20.498Z [INFO]  client.driver_mgr.docker: started container: driver=docker container_id=4862c4921af65432be66510be472808508ecdf23bbf52205d5a2e38702e91035

2021-09-29T16:32:20.512Z [DEBUG] client.driver_mgr.docker.docker_logger: starting plugin: driver=docker path=/usr/bin/nomad args=[/usr/bin/nomad, docker_logger]

2021-09-29T16:32:20.514Z [DEBUG] client.driver_mgr.docker.docker_logger: plugin started: driver=docker path=/usr/bin/nomad pid=18685

2021-09-29T16:32:20.515Z [DEBUG] client.driver_mgr.docker.docker_logger: waiting for RPC address: driver=docker path=/usr/bin/nomad

2021-09-29T16:32:20.525Z [DEBUG] client.driver_mgr.docker.docker_logger: using plugin: driver=docker version=2

2021-09-29T16:32:20.525Z [DEBUG] client.driver_mgr.docker.docker_logger.nomad: plugin address: driver=docker @module=docker_logger address=/tmp/plugin997128501 network=unix timestamp=2021-09-29T16:32:20.524Z

2021-09-29T16:32:20.527Z [DEBUG] client.driver_mgr.docker.docker_logger.nomad: using client connection initialized from environment: driver=docker @module=docker_logger timestamp=2021-09-29T16:32:20.527Z

2021-09-29T16:32:20.664Z [DEBUG] client: updated allocations: index=23 total=2 pulled=0 filtered=2

2021-09-29T16:32:20.666Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=2

2021-09-29T16:32:20.667Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=2 errors=0

2021-09-29T16:32:22.095Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:22.096Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.288487ms

2021-09-29T16:32:22.178Z [DEBUG] http: request complete: method=GET path=/v1/deployment/e83aaa10-073c-7aa0-f858-1a03e44362aa?index=20&stale=&wait=2000ms duration=2.108702013s

2021-09-29T16:32:24.291Z [DEBUG] http: request complete: method=GET path=/v1/deployment/e83aaa10-073c-7aa0-f858-1a03e44362aa?index=20&stale=&wait=2000ms duration=2.109113163s

2021-09-29T16:32:25.124Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:25.125Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.733815ms

2021-09-29T16:32:25.283Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44280

2021-09-29T16:32:25.852Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=394.642µs

2021-09-29T16:32:26.359Z [DEBUG] http: request complete: method=GET path=/v1/deployment/e83aaa10-073c-7aa0-f858-1a03e44362aa?index=20&stale=&wait=2000ms duration=2.062897681s

2021-09-29T16:32:28.146Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:28.146Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.047603ms

2021-09-29T16:32:28.420Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=521.464µs

2021-09-29T16:32:28.459Z [DEBUG] http: request complete: method=GET path=/v1/deployment/e83aaa10-073c-7aa0-f858-1a03e44362aa?index=20&stale=&wait=2000ms duration=2.094115784s

2021-09-29T16:32:30.529Z [DEBUG] http: request complete: method=GET path=/v1/deployment/e83aaa10-073c-7aa0-f858-1a03e44362aa?index=20&stale=&wait=2000ms duration=2.063291766s

2021-09-29T16:32:30.721Z [DEBUG] http: request complete: method=GET path=/v1/deployment/e83aaa10-073c-7aa0-f858-1a03e44362aa?index=20&stale=&wait=2000ms duration=187.38973ms

2021-09-29T16:32:30.723Z [DEBUG] client: updated allocations: index=24 total=2 pulled=0 filtered=2

2021-09-29T16:32:30.726Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=2

2021-09-29T16:32:30.727Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=2 errors=0

2021-09-29T16:32:31.417Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:31.419Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.788799ms

2021-09-29T16:32:31.976Z [DEBUG] worker: dequeued evaluation: eval_id=6bf1907e-fb0c-9e55-8989-263e96efa607

2021-09-29T16:32:31.978Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=6bf1907e-fb0c-9e55-8989-263e96efa607 job_id=bridged namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0)

Deployment Update for ID "e83aaa10-073c-7aa0-f858-1a03e44362aa": Status "successful"; Description "Deployment completed successfully"

Desired Changes for "bridged": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)"

2021-09-29T16:32:31.983Z [DEBUG] http: request complete: method=GET path=/v1/deployment/e83aaa10-073c-7aa0-f858-1a03e44362aa?index=24&stale=&wait=2000ms duration=1.25474607s

2021-09-29T16:32:31.983Z [DEBUG] worker: submitted plan for evaluation: eval_id=6bf1907e-fb0c-9e55-8989-263e96efa607

2021-09-29T16:32:31.984Z [DEBUG] worker.service_sched: setting eval status: eval_id=6bf1907e-fb0c-9e55-8989-263e96efa607 job_id=bridged namespace=default status=complete

2021-09-29T16:32:31.990Z [DEBUG] worker: updated evaluation: eval="<Eval "6bf1907e-fb0c-9e55-8989-263e96efa607" JobID: "bridged" Namespace: "default">"

2021-09-29T16:32:31.990Z [DEBUG] worker: ack evaluation: eval_id=6bf1907e-fb0c-9e55-8989-263e96efa607

2021-09-29T16:32:34.441Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:34.442Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.039788ms

2021-09-29T16:32:35.285Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44296

2021-09-29T16:32:35.859Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=446.078µs

2021-09-29T16:32:37.464Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:37.465Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.297617ms

2021-09-29T16:32:38.426Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=361.583µs

2021-09-29T16:32:40.487Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:40.488Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.55922ms

2021-09-29T16:32:43.242Z [DEBUG] http: request complete: method=GET path=/v1/job/bridged/allocations?all=false duration=393.562µs

2021-09-29T16:32:43.244Z [DEBUG] http: request complete: method=GET path=/v1/allocation/1a3e8ff1-ca08-633e-42dd-222b44c60734?namespace=default duration=416.733µs

2021-09-29T16:32:43.247Z [DEBUG] http: request complete: method=GET path=/v1/node/23b96b94-8b8e-92fa-4309-06da749bc6d6 duration=328.076µs

2021-09-29T16:32:43.249Z [INFO]  client: task exec session starting: exec_id=61a832a1-e155-56b7-0b37-ecc7c076b206 alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged command=[/bin/sh, -c, "nslookup nomad && echo "--------" && cat /etc/resolv.conf"] tty=true access_token_name= access_token_id=

2021-09-29T16:32:43.465Z [INFO]  client: task exec session ended: exec_id=61a832a1-e155-56b7-0b37-ecc7c076b206

2021-09-29T16:32:43.465Z [DEBUG] http: alloc exec channel closed with error: error="websocket: close 1006 (abnormal closure): unexpected EOF"

2021-09-29T16:32:43.466Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/1a3e8ff1-ca08-633e-42dd-222b44c60734/exec?command=%5B%22%2Fbin%2Fsh%22%2C%22-c%22%2C%22nslookup+nomad+%5Cu0026%5Cu0026+echo+%5C%22--------%5C%22+%5Cu0026%5Cu0026+cat+%2Fetc%2Fresolv.conf%22%5D&region=global&task=bridged&tty=true duration=216.570833ms

2021-09-29T16:32:43.503Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:43.504Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=536.73µs

2021-09-29T16:32:45.287Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44312

2021-09-29T16:32:45.872Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=563.063µs

2021-09-29T16:32:46.518Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:46.519Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.103852ms

2021-09-29T16:32:48.430Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=279.671µs

2021-09-29T16:32:49.541Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:49.542Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.800436ms

2021-09-29T16:32:52.563Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:52.569Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=6.46909ms

2021-09-29T16:32:55.289Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44320

2021-09-29T16:32:55.587Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:55.588Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.151653ms

2021-09-29T16:32:55.878Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=331.331µs

2021-09-29T16:32:58.434Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=234.995µs

2021-09-29T16:32:58.600Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:32:58.600Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=502.108µs

2021-09-29T16:33:01.613Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:01.614Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=567.558µs

2021-09-29T16:33:04.629Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:04.631Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=2.938453ms

2021-09-29T16:33:05.300Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44328

2021-09-29T16:33:05.882Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=414.251µs

2021-09-29T16:33:07.650Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:07.652Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.873631ms

2021-09-29T16:33:08.437Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=652.088µs

2021-09-29T16:33:10.671Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:10.672Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.576884ms

2021-09-29T16:33:13.691Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:13.691Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.034086ms

2021-09-29T16:33:15.303Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44336

2021-09-29T16:33:15.891Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=453.37µs

2021-09-29T16:33:16.712Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:16.714Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.814705ms

2021-09-29T16:33:18.451Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=355.355µs

2021-09-29T16:33:19.737Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:19.738Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.519725ms

2021-09-29T16:33:22.760Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:22.761Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.743603ms

2021-09-29T16:33:24.669Z [DEBUG] http: request complete: method=GET path=/v1/job/bridged/allocations?all=false duration=262.235µs

2021-09-29T16:33:24.671Z [DEBUG] http: request complete: method=GET path=/v1/allocation/1a3e8ff1-ca08-633e-42dd-222b44c60734?namespace=default duration=266.047µs

2021-09-29T16:33:24.674Z [DEBUG] http: request complete: method=GET path=/v1/node/23b96b94-8b8e-92fa-4309-06da749bc6d6 duration=234.913µs

2021-09-29T16:33:24.676Z [INFO]  client: task exec session starting: exec_id=dd927b0c-f0c1-c552-de99-b9623f32e5fd alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged command=[/bin/sh, -c, "cat /etc/resolv.conf && echo "--------" && nslookup nomad"] tty=true access_token_name= access_token_id=

2021-09-29T16:33:24.819Z [INFO]  client: task exec session ended: exec_id=dd927b0c-f0c1-c552-de99-b9623f32e5fd

2021-09-29T16:33:24.820Z [DEBUG] http: alloc exec channel closed with error: error=

2021-09-29T16:33:24.821Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/1a3e8ff1-ca08-633e-42dd-222b44c60734/exec?command=%5B%22%2Fbin%2Fsh%22%2C%22-c%22%2C%22cat+%2Fetc%2Fresolv.conf+%5Cu0026%5Cu0026+echo+%5C%22--------%5C%22+%5Cu0026%5Cu0026+nslookup+nomad%22%5D&region=global&task=bridged&tty=true duration=144.938719ms

2021-09-29T16:33:25.307Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44352

2021-09-29T16:33:25.782Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:25.783Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.566122ms

2021-09-29T16:33:25.917Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=356.255µs

2021-09-29T16:33:28.465Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=353.456µs

2021-09-29T16:33:28.803Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:28.804Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.197258ms

2021-09-29T16:33:31.699Z [DEBUG] http: request complete: method=GET path=/v1/job/nonbridged/allocations?all=false duration=215.042µs

2021-09-29T16:33:31.701Z [DEBUG] http: request complete: method=GET path=/v1/allocation/5c9fad52-2e79-e3b6-74b1-1414277695ce?namespace=default duration=252.585µs

2021-09-29T16:33:31.704Z [DEBUG] http: request complete: method=GET path=/v1/node/23b96b94-8b8e-92fa-4309-06da749bc6d6 duration=297.193µs

2021-09-29T16:33:31.707Z [INFO]  client: task exec session starting: exec_id=ca72ed9a-4a86-ef5e-9114-6f8ed67debf3 alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged command=[/bin/sh, -c, "cat /etc/resolv.conf && echo "--------" && nslookup nomad"] tty=true access_token_name= access_token_id=

2021-09-29T16:33:31.822Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:31.822Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=870.261µs

2021-09-29T16:33:31.837Z [INFO]  client: task exec session ended: exec_id=ca72ed9a-4a86-ef5e-9114-6f8ed67debf3

2021-09-29T16:33:31.838Z [DEBUG] http: alloc exec channel closed with error: error="websocket: close 1006 (abnormal closure): unexpected EOF"

2021-09-29T16:33:31.838Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/5c9fad52-2e79-e3b6-74b1-1414277695ce/exec?command=%5B%22%2Fbin%2Fsh%22%2C%22-c%22%2C%22cat+%2Fetc%2Fresolv.conf+%5Cu0026%5Cu0026+echo+%5C%22--------%5C%22+%5Cu0026%5Cu0026+nslookup+nomad%22%5D&region=global&task=nonbridged&tty=true duration=132.225334ms

2021-09-29T16:33:34.839Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:34.840Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.145476ms

2021-09-29T16:33:35.307Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44368

==> Caught signal: interrupt

2021-09-29T16:33:35.700Z [INFO]  agent: requesting shutdown

2021-09-29T16:33:35.700Z [INFO]  client: shutting down

2021-09-29T16:33:35.776Z [DEBUG] client: updated allocations: index=29 total=2 pulled=0 filtered=2

2021-09-29T16:33:35.923Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=399.932µs

2021-09-29T16:33:37.858Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:37.860Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=1.567499ms

2021-09-29T16:33:38.488Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=749.073µs

2021-09-29T16:33:40.873Z [DEBUG] http: request failed: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats error="Unknown allocation "454498d4-8a2e-e044-03ac-1507a55d3539"" code=404

2021-09-29T16:33:40.873Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/454498d4-8a2e-e044-03ac-1507a55d3539/stats duration=475.509µs

2021-09-29T16:33:40.880Z [INFO]  client.driver_mgr.docker: stopped container: container_id=4862c4921af65432be66510be472808508ecdf23bbf52205d5a2e38702e91035 driver=docker

2021-09-29T16:33:40.883Z [DEBUG] client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=18685

2021-09-29T16:33:40.883Z [DEBUG] client.driver_mgr.docker.docker_logger.stdio: received EOF, stopping recv loop: driver=docker err="rpc error: code = Unavailable desc = transport is closing"

2021-09-29T16:33:40.884Z [DEBUG] client.driver_mgr.docker.docker_logger: plugin exited: driver=docker

2021-09-29T16:33:40.890Z [DEBUG] client.driver_mgr.docker: image id reference count decremented: driver=docker image_id=sha256:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a references=1

2021-09-29T16:33:40.891Z [INFO]  client.gc: marking allocation for GC: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734

2021-09-29T16:33:40.893Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged path=/usr/bin/nomad pid=18636

2021-09-29T16:33:40.894Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin exited: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged

2021-09-29T16:33:40.894Z [DEBUG] client.alloc_runner.task_runner: task run loop exiting: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged

2021-09-29T16:33:40.894Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.stdio: received EOF, stopping recv loop: alloc_id=1a3e8ff1-ca08-633e-42dd-222b44c60734 task=bridged err="rpc error: code = Unavailable desc = transport is closing"

2021-09-29T16:33:40.945Z [INFO]  client.driver_mgr.docker: stopped container: container_id=32926edfa3458cb5e2c5d36e74fe3d034a00c6abc658754879b4c905696f1a81 driver=docker

2021-09-29T16:33:40.949Z [DEBUG] client.driver_mgr.docker.docker_logger.stdio: received EOF, stopping recv loop: driver=docker err="rpc error: code = Unavailable desc = transport is closing"

2021-09-29T16:33:40.950Z [DEBUG] client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=18469

2021-09-29T16:33:40.952Z [DEBUG] client.driver_mgr.docker.docker_logger: plugin exited: driver=docker

2021-09-29T16:33:40.957Z [DEBUG] client.driver_mgr.docker: image id reference count decremented: driver=docker image_id=sha256:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a references=0

2021-09-29T16:33:40.958Z [INFO]  client.gc: marking allocation for GC: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce

2021-09-29T16:33:40.960Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.stdio: received EOF, stopping recv loop: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged err="rpc error: code = Unavailable desc = transport is closing"

2021-09-29T16:33:40.961Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged path=/usr/bin/nomad pid=18366

2021-09-29T16:33:40.962Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin exited: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged

2021-09-29T16:33:40.962Z [DEBUG] client.alloc_runner.task_runner: task run loop exiting: alloc_id=5c9fad52-2e79-e3b6-74b1-1414277695ce task=nonbridged

2021-09-29T16:33:41.022Z [INFO]  client.plugin: shutting down plugin manager: plugin-type=device

2021-09-29T16:33:41.022Z [INFO]  client.plugin: plugin manager finished: plugin-type=device

2021-09-29T16:33:41.023Z [INFO]  client.plugin: shutting down plugin manager: plugin-type=driver

2021-09-29T16:33:41.023Z [INFO]  client.plugin: plugin manager finished: plugin-type=driver

2021-09-29T16:33:41.023Z [INFO]  client.plugin: shutting down plugin manager: plugin-type=csi

2021-09-29T16:33:41.023Z [INFO]  client.plugin: plugin manager finished: plugin-type=csi

2021-09-29T16:33:41.024Z [DEBUG] client.server_mgr: shutting down

2021-09-29T16:33:41.024Z [INFO]  nomad: shutting down server

2021-09-29T16:33:41.024Z [WARN]  nomad: serf: Shutdown without a Leave

2021-09-29T16:33:41.025Z [DEBUG] nomad: shutting down leader loop

2021-09-29T16:33:41.025Z [INFO]  nomad: cluster leadership lost

2021-09-29T16:33:41.028Z [ERROR] nomad.client: alloc update failed: error="raft is already shutdown"

2021-09-29T16:33:41.036Z [INFO]  agent: shutdown complete

2021-09-29T16:33:41.037Z [DEBUG] http: shutting down http server

The text was updated successfully, but these errors were encountered:

lgfa29 · 2021-10-02T01:58:02Z

Thanks for the report @jhaezebr.

I wonder if this is related to #11033, more specifically the last item:

with a network.dns block, both docker and exec tasks get the Nomad-managed resolv.conf generated by GenerateDNSMount

Could you try changing your host DNS configuration to see if it solves your problem?

jhaezebr · 2021-10-04T10:13:28Z

@lgfa29 I'm not sure if this was what you were asking, but it works when i add the following to the job configs:

  network {
      dns {
        searches = ["service.consul"]
      }
  }

output

$ nomad exec -job nonbridged /bin/sh -c 'cat /etc/resolv.conf && echo "--------" && nslookup nomad'
search service.consul
nameserver 10.0.0.2
nameserver 10.0.2.2
nameserver x.x.x.x
nameserver y.y.y.y
--------
Server:    10.0.0.2
Address 1: 10.0.0.2

Name:      nomad
Address 1: 10.0.2.15

$ nomad exec -job bridged /bin/sh -c 'cat /etc/resolv.conf && echo "--------" && nslookup nomad'
search service.consul
nameserver 10.0.0.2
nameserver 10.0.2.2
nameserver x.x.x.x
nameserver y.y.y.y
--------
Server:    10.0.0.2
Address 1: 10.0.0.2

Name:      nomad
Address 1: 10.0.2.15

job files

bridge.nomad

job "bridged" {
  datacenters = ["dc1"]
  group "bridged" {
    network {
      mode = "bridge"
      dns {
        searches = ["service.consul"]
      }
    }
    task "bridged" {
      driver = "docker"
      config {
        image = "busybox:1.28"
        dns_search_domains = ["service.consul"]
        command = "sleep"
        args = ["3600"]
      }
    }
  }
}

nonbridge.nomad

job "nonbridged" {
  datacenters = ["dc1"]
  group "nonbridged" {
    network {
      dns {
        searches = ["service.consul"]
      }
    }
    task "nonbridged" {
      driver = "docker"
      config {
        image = "busybox:1.28"
        dns_search_domains = ["service.consul"]
        command = "sleep"
        args = ["3600"]
      }
    }
  }
}

jhaezebr · 2021-10-04T10:20:25Z

Using the servers options the containers both contain the resolv.conf search option from the host

    network {
      dns {
        servers = ["10.0.0.2"]
      }
    }

resolv.conf in both of the containers:

search home
nameserver 10.0.0.2

So it seems that it is indeed the same problem?

jrasell · 2021-10-07T13:20:33Z

Hi @jhaezebr and thanks for raising this. I've taken a look into this today and wanted to share some my findings for you and other readers.

I firstly confirmed that just using vanilla Docker results in an expected resolv.conf file:

$ docker run --net=bridge --dns-search=service.consul busybox sleep 3600 &
$ docker exec 1494764cc5bd cat /etc/resolv.conf
search service.consul
nameserver 10.0.2.3

When running your example which uses bridge networking and inspecting the resolv.conf file, the below is detailed. Interestingly, the container ID detailed in the path, is not that of the busybox container, but that of the pause container that is used to secure the network namespace. This indicates to me that the container inherits the DNS configuration from the pause container, and therefore the dns_search_domains is ignored/overwritten by Docker when running in this configuration mode. It's worth noting, this setup is different to the initial Docker test which doesn't utilise network namespaces nor pause containers.

$ docker inspect 4f7c8ab8362c |jq '.[0].ResolvConfPath'
"/var/lib/docker/containers/9c877407dc33a2e84829faf04a0e72c74732df55ff7b4d25e4610ec83b14a624/resolv.conf"

$ docker ps
CONTAINER ID   IMAGE                                      COMMAND        CREATED         STATUS         PORTS     NAMES
4f7c8ab8362c   busybox:1.28                               "sleep 3600"   7 minutes ago   Up 7 minutes             bridged-0a72140d-6151-3506-ed85-c787a01b9959
9c877407dc33   gcr.io/google_containers/pause-amd64:3.1   "/pause"       7 minutes ago   Up 7 minutes             nomad_init_0a72140d-6151-3506-ed85-c787a01b9959

The Nomad job specification allows for setting of DNS configuration via the network stanza. Utilising this in the job specification included below, results in the desired resolv.conf.

job "bridged" {
  datacenters = ["dc1"]
  group "bridged" {
    network {
      mode = "bridge"
      dns {
        searches = ["service.consul"]
      }
    }
    task "bridged" {
      driver = "docker"
      config {
        image = "busybox:1.28"
        command = "sleep"
        args = ["3600"]
      }
    }
  }
}

$ nomad alloc exec cb1b2d32 cat /etc/resolv.conf
search service.consul
nameserver 10.0.2.3

I will discuss this internally and would be interested in your feedback on the approach; but I feel using the network.dns stanza is the preferred method when using Nomad with Docker in bridge mode and network namespaces. We could mark the dns_search_domains Docker configuration parameter as such, indicating the exact use cases and alternatives. An alternative approach to a resolution would be to patch the creation of the pause container so that the dns_search_domains is passed to this container creation. I feel this is a suboptimal fix though as dns_search_domains can be set per Docker task in bridge mode, however, the pause container is set once per network namespace and thus Nomad job task group.

jhaezebr · 2021-10-08T07:41:27Z

Hi @jrasell ,
I just tried and apparently the network.dns also works in the nonbridged docker container, even with multiple task per group. So to me it seems that network.dns would be the preferred choice to configure any job, and keep dns_search_domains to the special cases where some containers need to have a special configuration? As you suggested the documentation could then indicate that this dns_search_domains option does not work in bridged network mode.

In any case, I'd like to thank you and @lgfa29 for the help on this problem.

tgross · 2022-03-08T16:33:20Z

Doc fix in #12229

jhaezebr added the type/bug label Sep 29, 2021

lgfa29 added stage/needs-investigation theme/driver/docker theme/networking labels Oct 2, 2021

tgross mentioned this issue Mar 8, 2022

docs: add note about docker DNS config when using bridge mode #12229

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dns_search_domain not applied when using bridge network mode #11239

dns_search_domain not applied when using bridge network mode #11239

jhaezebr commented Sep 29, 2021

lgfa29 commented Oct 2, 2021

jhaezebr commented Oct 4, 2021

jhaezebr commented Oct 4, 2021

jrasell commented Oct 7, 2021

jhaezebr commented Oct 8, 2021

tgross commented Mar 8, 2022

dns_search_domain not applied when using bridge network mode #11239

dns_search_domain not applied when using bridge network mode #11239

Comments

jhaezebr commented Sep 29, 2021

Nomad version

Operating system and Environment details

Issue

Reproduction steps

Expected Result

Actual Result

Job file (if appropriate)

Nomad logs

lgfa29 commented Oct 2, 2021

jhaezebr commented Oct 4, 2021

jhaezebr commented Oct 4, 2021

jrasell commented Oct 7, 2021

jhaezebr commented Oct 8, 2021

tgross commented Mar 8, 2022