Container.exec hangs unexpectedly

[rwcarlsen]

I have a k8s charm I've been tinkering with (github.com/rwcarlsen/icecream-k8s-operator). In it, I call process = container.exec(['apt', 'install', '-y', 'foo']) to install a couple packages via apt on the workload container. When I do this process.wait() unexpectedly hangs indefinitely. The charm/ops traceback info when things are hung is:

######### ProcessID=149, ThreadID=140318605281088 #########
File: "./src/charm.py", line 181, in <module>
  main(IcecreamK8SOperatorCharm)
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/main.py", line 431, in main
  _emit_charm_event(charm, dispatcher.event_name)
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
  event_to_emit.emit(*args, **kwargs)
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/framework.py", line 283, in emit
  framework._emit(event)
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/framework.py", line 743, in _emit
  self._reemit(event_path)
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/framework.py", line 790, in _reemit
  custom_handler(event)
File: "./src/charm.py", line 37, in _on_icecream_pebble_ready
  self._install_workload(container)
File: "./src/charm.py", line 95, in _install_workload
  process.wait()
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/pebble.py", line 1018, in wait
  exit_code = self._wait()
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/pebble.py", line 1028, in _wait
  change = self._client.wait_change(self._change_id, timeout=timeout)
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/pebble.py", line 1505, in wait_change
  return self._wait_change_using_wait(change_id, timeout)
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/pebble.py", line 1526, in _wait_change_using_wait
  return self._wait_change(change_id, this_timeout)
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/pebble.py", line 1541, in _wait_change
  resp = self._request('GET', '/v1/changes/{}/wait'.format(change_id), query)
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/pebble.py", line 1281, in _request
  response = self._request_raw(method, path, query, headers, data)
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/pebble.py", line 1315, in _request_raw
  response = self.opener.open(request, timeout=self.timeout)
File: "/usr/lib/python3.8/urllib/request.py", line 525, in open
  response = self._open(req, data)
File: "/usr/lib/python3.8/urllib/request.py", line 542, in _open
  result = self._call_chain(self.handle_open, protocol, protocol +
File: "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
  result = func(*args)
File: "/var/lib/juju/agents/unit-icecream-k8s-operator-0/charm/venv/ops/pebble.py", line 84, in http_open
  return self.do_open(_UnixSocketConnection, req, socket_path=self.socket_path)
File: "/usr/lib/python3.8/urllib/request.py", line 1358, in do_open
  r = h.getresponse()
File: "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
  response.begin()
File: "/usr/lib/python3.8/http/client.py", line 316, in begin
  version, status, reason = self._read_status()
File: "/usr/lib/python3.8/http/client.py", line 277, in _read_status
  line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File: "/usr/lib/python3.8/socket.py", line 669, in readinto
  return self._sock.recv_into(b)

I still don't understand exactly what is causing the hang. But when I do container.exec(['apt', 'install', '-y', 'foo'], stderr=sys.stderr, stdout=sys.stdout) (adding the stdout/err kwargs), it doesn't hang and works as expected. @benhoyt - any sense for what/where the culprit might be?

[benhoyt]

Interesting, thanks for the report. Hmm, no, no real sense without digging in. However, I did just run into a very similar issue with os/exec in Go on a side project the other day (see golang/go#23019). One of the workarounds was to explicitly set stdout/stderr to os.Stdout and os.Stderr. It had to do with a child process holding the pipes open. Anyway, probably just an interesting coincidence.

[pengale]

@rwcarlsen have you run into this issue since? Do we have a path forward to maybe addressing and fixing this?

[rwcarlsen]

It's still a problem as far as I know although I haven't checked it recently. It was a very reproduceable/persistent problem that seems unlikely to just go away without someone coming up with a direct fix. I just haven't made it a priority to investigate further.

[benhoyt]

We run a bunch of automated exec tests against real Pebble in CI and don't have any issues there. Closing this for now unless and until we have a recent, specific repro case.