Problem installing or searching pack, temporary failure in name resolution

[JustDevZero]

Describe the problem
A clear and concise description of what the problem is.

Versions

• Host OS: Fedora 33 And Ubuntu 18.04
• docker: Docker version 20.10.3, build 48d30b5
• docker-compose: docker-compose version 1.28.2, build 67630359
• stackstorm version: v3.3.0

To Reproduce
If necessary, please specify steps to reproduce the behavior.

Login to st2client with:

docker-compose exec st2client bash

And then try to search or install.

It will fail with:

st2 pack search datadog

root@554d99419ce3:/opt/stackstorm# st2 pack search docker
ERROR: 400 Client Error: Bad Request
MESSAGE: No results from the index: tried https://index.stackstorm.org/v1/index.json.
Status: [
    {
        "url": "https://index.stackstorm.org/v1/index.json",
        "packs": 0,
        "message": "ConnectionError(MaxRetryError(\"HTTPSConnectionPool(host='index.stackstorm.org', port=443): Max retries exceeded with url: /v1/index.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f448e565710>: Failed to establish a new connection: [Errno -3] Lookup timed out',))\",),)",
        "error": "unresponsive"
    }
] for url: http://st2api:9101/packs/index/search



root@554d99419ce3:/opt/stackstorm# st2 pack install datadog

	[ succeeded ] init_task
	[  failed   ] download_pack

id: 6038397af6cde4fdb05d5c08
action.ref: packs.install
parameters: 
  packs:
  - datadog
  python3: false
status: failed
start_timestamp: Thu, 25 Feb 2021 23:57:46 UTC
end_timestamp: Thu, 25 Feb 2021 23:57:59 UTC
result: 
  errors:
  - message: Execution failed. See result for details.
    result:
      exit_code: 1
      result: None
      stderr: "Index parsing error: {
    "url": "https://index.stackstorm.org/v1/index.json",
    "packs": 0,
    "message": "ConnectionError(MaxRetryError(\\"HTTPSConnectionPool(host='index.stackstorm.org', port=443): Max retries exceeded with url: /v1/index.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8ff5cdc18>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))\\",),)",
    "error": "unresponsive"
}
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 334, in <module>
    obj.run()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 193, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 90, in run
    return self._validate_result(result=result, repo_url=pack_url)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 111, in _validate_result
    raise Exception(message)
Exception: The pack has not been downloaded from "None".

Errors:
No results from the index: tried https://index.stackstorm.org/v1/index.json.
Status: [
    {
        "url": "https://index.stackstorm.org/v1/index.json",
        "packs": 0,
        "message": "ConnectionError(MaxRetryError(\\"HTTPSConnectionPool(host='index.stackstorm.org', port=443): Max retries exceeded with url: /v1/index.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8ff5cdc18>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))\\",),)",
        "error": "unresponsive"
    }
]
"
      stdout: ''
    task_id: download_pack
    type: error
  output:
    conflict_list: null
    message: ''
    packs_list: null
    warning_list: null
+--------------------------+------------------------+---------------+----------------+-------------------------------+
| id                       | status                 | task          | action         | start_timestamp               |
+--------------------------+------------------------+---------------+----------------+-------------------------------+
| 6038397b1ac1577e7e12dae3 | succeeded (0s elapsed) | init_task     | core.noop      | Thu, 25 Feb 2021 23:57:47 UTC |
| 6038397c1ac1577e7e12daf3 | failed (11s elapsed)   | download_pack | packs.download | Thu, 25 Feb 2021 23:57:48 UTC |

If try to do some curl... from st2api container:

root@2bc662f52cab:/opt/stackstorm# curl https://index.stackstorm.org/v1/index.json|wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0curl: (6) Could not resolve host: index.stackstorm.org
0

Modifying resolve.conf for adding 1.1.1.1 after docker resolution improves the thing but still.

curl https://index.stackstorm.org/v1/index.json|wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  678k    0  678k    0     0   406k      0 --:--:--  0:00:02 --:--:--  406k
16298

From s2client container on the other hand it's inmediate.

curl https://index.stackstorm.org/v1/index.json|wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  678k    0  678k    0     0  3879k      0 --:--:-- --:--:-- --:--:-- 3857k
16298

For some reason, s2api container it's taking more time to resolv... But even with that it's no reason for getting a Temporary failure in name resolution.

Am i installing them wrong?

[minsis]

By default bridge networks dont forward to the outside world and need to have additional parameters set on the host for this to work correctly.

https://docs.docker.com/network/bridge/#enable-forwarding-from-docker-containers-to-the-outside-world

[JustDevZero]

@minsis I tried it, and indeed also tried changing the type of network, but the problem is still the same.

[minsis]

@JustDevZero We also faced some of these issues (not with StackStorm) when ipv6 was enabled and ultimately disabled it. Do you also have ipv6 enabled?

[JustDevZero]

I'll check it as soon as I'm at my office hours, but I think you're maybe having a point on the ipv6, AFAIK we have ipv6 disabled on whole network. El jue., 4 mar. 2021 22:34, Dennis Whitney <notifications@github.com> escribió:

…

@JustDevZero <https://github.com/JustDevZero> We also faced some of these issues (not with StackStorm) when ipv6 was enabled and ultimately disabled it. Do you also have ipv6 enabled? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#216 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABKFV33A6MSV3UXJOM4ZJTTB74GVANCNFSM4YHOXTIQ> .

[minsis]

@JustDevZero The network itself can have ipv6 disabled as a whole but that doesn't prevent localhost from having it enabled. Every device has its own link local ipv6 address when it comes up.

[JustDevZero]

It looks that it's ipv6 fault then, host have ipv6 disabled but somehow docker it's still trying to get ipv6 not 4. So... I guess it's matter of dealing with docker config, maybe?

[daniel-rocha-ciandt]

Hi,
I'm also facing the same issue, from curl it works perfectly.

Every action in python 3 that need to resolve a DNS happens the same problem:

@JustDevZero have you manager to find out what was the root cause of this issue?

Versions

• Host OS: Ubuntu 20.04
• docker: Docker version 20.10.7, build 20.10.7-0ubuntu1~20.04.1
• docker-compose version 1.29.1, build c34c88b2
• stackstorm version: v3.5.0
• Python: v3.6.9

[cydergoth]

This fixed the issue for me:

diff --git a/docker-compose.yml b/docker-compose.yml
index 3a05ac0..b1c47fd 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -2,6 +2,9 @@ version: '3'

 services:
   st2web:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2web:${ST2_VERSION:-latest}
     restart: on-failure
     environment:
@@ -42,6 +45,9 @@ services:
     dns_search: .
     command: /makesecrets.sh
   st2api:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2api:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -65,6 +71,9 @@ services:
       - ${ST2_PACKS_DEV:-./packs.dev}:/opt/stackstorm/packs.dev:rw
     dns_search: .
   st2stream:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2stream:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -76,6 +85,9 @@ services:
       - ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
     dns_search: .
   st2scheduler:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2scheduler:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -88,6 +100,9 @@ services:
       - ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
     dns_search: .
   st2workflowengine:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2workflowengine:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -101,6 +116,9 @@ services:
       - stackstorm-keys:/etc/st2/keys:ro
     dns_search: .
   st2auth:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2auth:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -113,6 +131,9 @@ services:
       - ./files/htpasswd:/etc/st2/htpasswd:ro
     dns_search: .
   st2actionrunner:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2actionrunner:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -133,6 +154,9 @@ services:
       - stackstorm-keys:/etc/st2/keys:ro
     dns_search: .
   st2garbagecollector:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2garbagecollector:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -144,6 +168,9 @@ services:
       - ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
     dns_search: .
   st2notifier:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2notifier:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -156,6 +183,9 @@ services:
       - ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
     dns_search: .
   st2rulesengine:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2rulesengine:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -167,6 +197,9 @@ services:
       - ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
     dns_search: .
   st2sensorcontainer:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2sensorcontainer:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -182,6 +215,9 @@ services:
       - stackstorm-packs-configs:/opt/stackstorm/configs:ro
       - ${ST2_PACKS_DEV:-./packs.dev}:/opt/stackstorm/packs.dev:ro
   st2timersengine:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2timersengine:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -192,6 +228,9 @@ services:
     volumes:
       - ./files/st2.docker.conf:/etc/st2/st2.docker.conf:ro
   st2client:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2actionrunner:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -222,6 +261,9 @@ services:
       - ./files/st2-cli.conf:/root/.st2/config
       - ./scripts/st2client-startup.sh:/st2client-startup.sh
   st2chatops:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2chatops:${ST2_VERSION:-latest}
     restart: on-failure:5
     depends_on:

[minsis]

By default ipv4 forwarding is not enabled. This is mostly where the issue is with getting traffic to leave the container.

Add net.ipv4.ip_forward=1 to /etc/sysctl.conf to fix it.