Google API client breaks name resolution for paho in a docker context

[wsw70]

My (home grade) code accesses the Google Calendar API and then publishes information it finds there to a MQTT bus.

It used to run as a service on my server and I decided to move it into a docker container (also on my server).

The problem I encounter is that calling the Google API libraries apparently breaks name resolution in paho when ran in a docker container.

The following code works OK when started from the command line of my server (keys and endpoints redacted)

import googleapiclient.discovery
import paho.mqtt.publish
from google.oauth2 import service_account as google_oauth2_service_account

paho.mqtt.publish.single(
    topic=f"testtopic",
    payload="before google",
    retain=True,
    hostname='mqtt.mydomain'
)
print("before google")

google_credentials = google_oauth2_service_account.Credentials.from_service_account_info(
    {
        "type": "service_account",
        "project_id": "myprojectid",
        "private_key_id": "8...4",
        "private_key": "-----BEGIN PRIVATE KEY-----\nM....Kag==\n-----END PRIVATE KEY-----\n",
        "client_email": "smthg@smthg.iam.gserviceaccount.com",
        "client_id": "1...9",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/XXX%40dXXX.iam.gserviceaccount.com"
    },
    scopes=['https://www.googleapis.com/auth/calendar'],
    subject='XXX@YYYY'
)
google_service = googleapiclient.discovery.build('calendar', 'v3', credentials=google_credentials, cache_discovery=False)
print("after google")

paho.mqtt.publish.single(
    topic=f"testtopic",
    payload="after google",
    retain=True,
    hostname='mqtt.mydomain'
)
print("before end of script")

In an MQTT explorer I see that the topic has been published to twice, with before google and after google, and the output is

before google
after google
end of script

So far, so good.

I then moved that code file into docker via the following Dockerfile:

FROM ubuntu:20.04
RUN apt update && DEBIAN_FRONTEND=noninteractive apt install -y python3-arrow python3-pip python3-paho-mqtt
RUN pip3 install google-api-python-client oauth2client
WORKDIR /app
COPY . .
CMD python3 googlecalendartest.py ; sleep 1000

The build went fine but the program crashes with (this is the only log)

before google
after google
Traceback (most recent call last):
  File "googlecalendartest.py", line 32, in <module>
    paho.mqtt.publish.single(
  File "/usr/lib/python3/dist-packages/paho/mqtt/publish.py", line 227, in single
    multiple([msg], hostname, port, client_id, keepalive, will, auth, tls,
  File "/usr/lib/python3/dist-packages/paho/mqtt/publish.py", line 163, in multiple
    client.connect(hostname, port, keepalive)
  File "/usr/lib/python3/dist-packages/paho/mqtt/client.py", line 937, in connect
    return self.reconnect()
  File "/usr/lib/python3/dist-packages/paho/mqtt/client.py", line 1071, in reconnect
    sock = self._create_socket_connection()
  File "/usr/lib/python3/dist-packages/paho/mqtt/client.py", line 3522, in _create_socket_connection
    return socket.create_connection(addr, source_address=source, timeout=self._keepalive)
  File "/usr/lib/python3.8/socket.py", line 787, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

I do see that testtopic received a payload of before google, but the one after the Google API calls is not there anymore.

socket.gaierror: [Errno -2] Name or service not known suggests that mqtt.mydomain is not resolvable (I think).

I added a print(requests.get("http://google.com").status_code) call after the Google API part and it works, so resolution is fine.

It means that the resolution is broken only for paho MQTT.

[BertKleewein]

I have a bug with a completely different context, a completely different setup, but the same symptoms. e.g. Paho sees a socket.gaierror: [Errno -2] Name or service not know but other processes are able to resolve the name.

My issue: Azure/azure-iot-sdk-python#548

@wsw70 - how easily can you repro this issue?

Correction: in your example, other objects in the current process are also able to resolve names, so it may be slightly different.

[BertKleewein]

I don't think this is related to my issue after all. Or, at least, it's not directly related. socket.getaddrinfo seems to be widely regarded as buggy and broken, and we're both seeing problems with socket.getaddrinfo returning failure.

For your problem, I've been looking at the requests code to see why it might be succeeding, and I can see that it's being much more specific about the parameters it passes to getaddrinfo. I can't say why requests is succeeding and Paho is failing, but the bug reports are numerous enough and strange enough that I wouldn't be surprised by anything.

As to why this is broken in Docker and not outside docker, it's very possible that your outside-docker DNS resolver is different from your inside-docker DNS resolver. One possible way to fix your issue is to install nscd into your container. This should cache DNS entries and make this error less likely. reference: https://stackoverflow.com/a/19930436/6262999

[MattBrittan]

I'm going to close this off due to age/inactivity.

If you are still having issues (with the latest client) then please feel free to reopen. However it would be useful if you could confirm that mqtt.mydomain resolves within docker (as this may well be a docker setup issue; particularly if mqtt.mydomain only resolves on a local DNS server).