Host name resolution error on MacOS

[elliotfontaine]

Versions

• Cylc: 8.2.4
• System: MacOS Monterey (12.7.4)

Description

On MacOS, local workflow schedulers sometimes stop responding, because of an host resolution error. It usually resolves itself after some time, and the workflow run starts responding again.

cylc stop bioreactor-workflow/run20

Traceback (most recent call last):
  File "/Users/elliotfontaine/mambaforge/envs/cylc/bin/cylc", line 10, in <module>
    sys.exit(main())
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/scripts/cylc.py", line 660, in main
    execute_cmd(command, *cmd_args)
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/scripts/cylc.py", line 286, in execute_cmd
    entry_point.resolve()(*args)
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/terminal.py", line 232, in wrapper
    wrapped_function(*wrapped_args, **wrapped_kwargs)
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/scripts/stop.py", line 275, in main
    rets = call_multi(
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/network/multi.py", line 29, in call_multi
    return asyncio.run(call_multi_async(*args, **kwargs))
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/network/multi.py", line 88, in call_multi_async
    async for (workflow_id, *args), result in unordered_map(
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/async_util.py", line 460, in unordered_map
    yield task._args, task.result()
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/scripts/stop.py", line 223, in run
    pclient = get_client(workflow_id, timeout=options.comms_timeout)
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/network/client_factory.py", line 63, in get_client
    return get_runtime_client(get_comms_method(), workflow, timeout=timeout)
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/network/client_factory.py", line 57, in get_runtime_client
    return WorkflowRuntimeClient(workflow, timeout=timeout)
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/network/client.py", line 241, in __init__
    WorkflowRuntimeClientBase.__init__(self, workflow, host, port, timeout)
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/network/client.py", line 71, in __init__
    host, port, _ = get_location(workflow)
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/network/__init__.py", line 84, in get_location
    host = get_fqdn_by_host(host)
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/hostuserutil.py", line 265, in get_fqdn_by_host
    return HostUtil.get_inst().get_fqdn_by_host(target)
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/hostuserutil.py", line 169, in get_fqdn_by_host
    if not self.is_remote_host(target):
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/hostuserutil.py", line 210, in is_remote_host
    host_info != self._get_host_info())
  File "/Users/elliotfontaine/mambaforge/envs/cylc/lib/python3.9/site-packages/cylc/flow/hostuserutil.py", line 135, in _get_host_info
    self._host_exs[target] = socket.gethostbyname_ex(target)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known: '1.0.0.127.in-addr.arpa'

cylc tui bioreactor-workflow/run20

[Errno 8] nodename nor servname provided, or not known: '1.0.0.127.in-addr.arpa'

Reproducible Example

I didn't find a way to accurately replicate the issue, from my pov it seems to happen at random.

[oliver-sanders]

This issue is caused by a long standing Python bug on Mac OS: python/cpython#49254

We've hacked around it, but our hack isn't holding up against your setup, sorry.

[oliver-sanders]

This (aborted) pull request would extend the hack to cover the address you have reported above: #5411

I aborted this pull request as I was unable to reproduce the setup / circumstances under which that address is returned.

If you have the ability to modify your installation, you can try out this patch (single file, couple of lines).

[oliver-sanders]

Closed by #5411