Startup in k8s results in error unless extra config applied

[lenalebt]

Describe the bug
I am playing around with the software in a kubernetes environment. Doing so, I stumbled upon a problem of the used asyncio-library: aio-libs/aiohttp#4554

This is the stack trace I am seeing:

/usr/local/lib/python3.8/asyncio/base_events.py:641: RuntimeWarning: coroutine 'Queue.get' was never awaited
  self._ready.clear()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "/virtualenv/bin/query-exporter", line 10, in <module>
    sys.exit(script())
  File "/virtualenv/lib/python3.8/site-packages/toolrack/script.py", line 105, in __call__
    self.main(parsed_args)
  File "/virtualenv/lib/python3.8/site-packages/prometheus_aioexporter/script.py", line 138, in main
    exporter.run()
  File "/virtualenv/lib/python3.8/site-packages/prometheus_aioexporter/web.py", line 70, in run
    run_app(
  File "/virtualenv/lib/python3.8/site-packages/aiohttp/web.py", line 419, in run_app
    loop.run_until_complete(_run_app(app,
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/virtualenv/lib/python3.8/site-packages/aiohttp/web.py", line 359, in _run_app
    await site.start()
  File "/virtualenv/lib/python3.8/site-packages/aiohttp/web_runner.py", line 100, in start
    self._server = await loop.create_server(  # type: ignore
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1463, in create_server
    raise OSError(err.errno, 'error while attempting '
OSError: [Errno 99] error while attempting to bind on address ('::1', 9560, 0, 0): cannot assign requested address

It seems to try to bind to an IPv6 address, although this may not be available in the environment.

I do have a workaround: start the application using the parameters -H 0.0.0.0 to force it to using IPv4. I found the possibility by having a look at the documentation of the underlying https://github.com/albertodonato/prometheus-aioexporter.

While I do think that this issue is not directly related to this project, I wanted to raise this bug (even if it is rejected) to document this workaround for others that may run into this issue.

Installation details

• Using Kubernetes 1.15 in GCloud
• Using docker image of version 2.2.0 of the query-exporter

[albertodonato]

Hi, could you please paste the output running the image passing -L DEBUG, specifically the message about "Listening on..." ?

The official docker image does pass -H 0.0.0.0 to only listen on IPv4:

query-exporter/Dockerfile

Line 33 in 1f2d38e

ENTRYPOINT ["query-exporter", "-H", "0.0.0.0"]

but perhaps something is not working correctly.

[ShireCiel]

same issue。
work fine with ["-H", "0.0.0.0"]

[albertodonato]

Can you provide the exact docker command line you're using?

If I run docker run -p 9560:9560/tcp -v "$PWD/config.yaml:/config.yaml" --rm -it adonato/query-exporter -- /config.yaml

a ps does show me the option being passed:

584622 pts/0    Ssl+   0:01 /virtualenv/bin/python3 /virtualenv/bin/query-exporter -H 0.0.0.0 -- /config.yaml

[lenalebt]

So, the exact command definition is command: ['query-exporter', '/config/config.yaml', '-H', '0.0.0.0'] for me (inside a k8s deployment file, which translates to query-exporter /config/config.yaml -H 0.0.0.0).

But since I do not have direct access to the cluster nodes themselves, I cannot provide a docker ps output.

EDIT: sorry, will paste with debug output later on

[lenalebt]

Running like this: command: ['query-exporter', '/config/config.yaml', '-L', 'DEBUG']

2020-04-20 13:00:42,692 - INFO - aiohttp.web - Listening on http://localhost:9560
2020-04-20 13:00:42,745 - DEBUG - query-exporter - connected to database "xxx"
2020-04-20 13:00:42,745 - DEBUG - query-exporter - running query "query1" on database "xxx"
2020-04-20 13:00:42,746 - DEBUG - query-exporter - running query "query2" on database "xxx"
Traceback (most recent call last):
  File "/virtualenv/bin/query-exporter", line 10, in <module>
    sys.exit(script())
  File "/virtualenv/lib/python3.8/site-packages/toolrack/script.py", line 105, in __call__
    self.main(parsed_args)
  File "/virtualenv/lib/python3.8/site-packages/prometheus_aioexporter/script.py", line 138, in main
    exporter.run()
  File "/virtualenv/lib/python3.8/site-packages/prometheus_aioexporter/web.py", line 70, in run
    run_app(
  File "/virtualenv/lib/python3.8/site-packages/aiohttp/web.py", line 419, in run_app
    loop.run_until_complete(_run_app(app,
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/virtualenv/lib/python3.8/site-packages/aiohttp/web.py", line 359, in _run_app
    await site.start()
  File "/virtualenv/lib/python3.8/site-packages/aiohttp/web_runner.py", line 100, in start
    self._server = await loop.create_server(  # type: ignore
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1463, in create_server
    raise OSError(err.errno, 'error while attempting '
OSError: [Errno 99] error while attempting to bind on address ('::1', 9560, 0, 0): cannot assign requested address

[albertodonato]

I'll add a note to the readme.

This is not really a bug of the app, though. I suspect that the reason it tries to bind ::1 as well is that localhost (which is the default value for -H) resolves to both 127.0.0.1 and ::1, so the app tries to bind ports on both addresses.

Explicitly passing 0.0.0.0 makes it only bind IPv4.
The official docker image does the same since Docker by default doesn't enable IPv6.

[lenalebt]

Thanks, note in the readme totally is fine :). Interesting though that I "worked around" the defined entrypoint as I was using the docker image you provided. That was not really my intention and I expected the k8s "command" to not do that 🤷‍♀️

[patilpravin557]

Hi,

In K8s i'm facing below issue after running the pod
what should be root cause of this?

kubectl logs queryexporter-pod
usage: query-exporter [-h] [-H HOST [HOST ...]] [-p PORT]
[-L {CRITICAL,ERROR,WARNING,INFO,DEBUG}]
[--process-stats] [-V] [--check-only]
config
query-exporter: error: the following arguments are required: config

[patilpravin557]

@lenalebt have you able to solve your issue, if yes what argument you have passed in pod definition file

[albertodonato]

Hi,

In K8s i'm facing below issue after running the pod
what should be root cause of this?

kubectl logs queryexporter-pod
usage: query-exporter [-h] [-H HOST [HOST ...]] [-p PORT]
[-L {CRITICAL,ERROR,WARNING,INFO,DEBUG}]
[--process-stats] [-V] [--check-only]
config
query-exporter: error: the following arguments are required: config

Could you provide the command line that gets passed to the docker image?
Also are you using the official image or your own?

[patilpravin557]

Hi,
I'm using pod defination yml file to run the query exporter, using official image

apiVersion: v1
kind: Pod
metadata:
name: queryexporter-pod
labels:
app: queryexporter

spec:
containers:
- name: exporter-container
image: adonato/query-exporter:latest
imagePullPolicy: Always

  volumeMounts:
    - name: config-volume
      mountPath: /etc/exporter/queryconfigmap.yml
      subPath: query-configmap.yml
  ports:
      - containerPort: 9560

volumes:
- name: config-volume
configMap:
name: query-configmap

[albertodonato]

You currently have to pass the config file name to the command line. I plan to change that in the next release to just always use the /config.yaml which gets mapped from the volume.

[patilpravin557]

you mean to say need to add this in pod definition file
command: ['query-exporter', '/config/config.yaml']

[albertodonato]

I'm not sure of the exact syntax, but yes, you have to pass the path of the config file as argument.