Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bridge_manager failed to push data due to TypeError #79

Open
tkriviradev opened this issue Mar 28, 2022 · 4 comments
Open

bridge_manager failed to push data due to TypeError #79

tkriviradev opened this issue Mar 28, 2022 · 4 comments

Comments

@tkriviradev
Copy link

tkriviradev commented Mar 28, 2022 •
edited
Loading

Hello,

I have exactly the same issues it happen after debian 10 > 11 upgrade
What I tried so far, I completely purged all old amplify agent configs directories and re-install.

Using how to (agent was reinstalled), after initial one min, agent start throwing error not able to reach https://receiver.amplify.nginx.com:443

Versions:

dpkg-query -l |grep nginx
nginx                                 1.20.2-1~buster                amd64        high performance web server
nginx-amplify-agent                   1.8.0-1~bullseye               amd64        Agent for NGINX Amplify monitoring platform
nginx-common                          1.10.3-1+deb9u3                all          small, powerful, scalable web/proxy server - common files

# Python
root@nginx:~# python3 --version
Python 3.9.2

# OS
 cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"

# Kernel
root@nginx:~# uname -a
Linux nginx 5.10.0-13-amd64 nginxinc/nginx-amplify-agent-priv#1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
2022-03-28 12:24:54,721 [9328] nginx_metrics [None] get http://127.0.0.1:80/nginx_status 200 10 106 0.004
2022-03-28 12:24:54,721 [9328] nginx_metrics 34d39fc314628b9ff20bfb12138a4c0cb407d6395fa5512cef844b95d9467663 collect in 0.006
2022-03-28 12:24:54,831 [9328] nginx_meta 34d39fc314628b9ff20bfb12138a4c0cb407d6395fa5512cef844b95d9467663 collect in 0.075
2022-03-28 12:24:55,016 [9328] supervisor system objects: ['e5389f19c285e6b27ce9f3cb150d4a97bad0ab4fa472deff1a5cc5c7194c6ae7']
2022-03-28 12:24:55,033 [9328] supervisor ps nginx output: ['   7533       1 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf', '   7534    7533 nginx: worker process', '   7535    7533 nginx: worker process', '   7536    7533 nginx: cache manager process', '']
2022-03-28 12:24:55,043 [9328] supervisor nginx objects: ['34d39fc314628b9ff20bfb12138a4c0cb407d6395fa5512cef844b95d9467663']
2022-03-28 12:24:55,044 [9328] supervisor status objects: []
2022-03-28 12:24:55,044 [9328] supervisor api objects: []
2022-03-28 12:24:55,674 [9328] bridge_manager failed POST "https://receiver.amplify.nginx.com:443/1.4/XXXXXXXXXXXXX/update/", exception: "HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0)"
2022-03-28 12:24:55,675 [9328] bridge_manager 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 114, in wrap_socket
    return self.sslsocket_class(
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 312, in __init__
    raise x
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 308, in __init__
    self.do_handshake()
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 672, in do_handshake
    self._wait(self._read_event, timeout_exc=_SSLErrorHandshakeTimeout)
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 151, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
socket.timeout: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 385, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/amplify/agent/common/util/http.py", line 94, in make_request
    r = self.session.post(
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0)
2022-03-28 12:24:55,765 [9328] bridge_manager 
2022-03-28 12:24:55,766 [9328] bridge_manager [None] post https://receiver.amplify.nginx.com:443/1.4/5147cd2d2fd569c76dbe47a18e2de225/update/ 500 1447 0 7.809
2022-03-28 12:24:55,766 [9328] bridge_manager http delay set to 10 (fails: 1)
2022-03-28 12:24:55,766 [9328] bridge_manager failed to push data due to ReadTimeout
2022-03-28 12:24:55,766 [9328] bridge_manager additional info:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 114, in wrap_socket
    return self.sslsocket_class(
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 312, in __init__
    raise x
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 308, in __init__
    self.do_handshake()
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 672, in do_handshake
    self._wait(self._read_event, timeout_exc=_SSLErrorHandshakeTimeout)
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 151, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
socket.timeout: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 385, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0)

Connectivity test

curl -v https://receiver.amplify.nginx.com
*   Trying 35.155.143.94:443...
* Connected to receiver.amplify.nginx.com (35.155.143.94) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=receiver.amplify.nginx.com
*  start date: Mar 14 21:05:25 2022 GMT
*  expire date: Jun 12 21:05:24 2022 GMT
*  subjectAltName: host "receiver.amplify.nginx.com" matched cert's "receiver.amplify.nginx.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: receiver.amplify.nginx.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Server: nginx
< Date: Mon, 28 Mar 2022 09:31:51 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
< X-Amplify-ID: 2017ea69c9f904f966ae0433d6f205a0
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host receiver.amplify.nginx.com left intact
@tkriviradev
Copy link
Author

UP^

@nkashiv
Copy link
Contributor

nkashiv commented Apr 14, 2022

@tkriviradev Please can you update your agent and check if that resolves this issue?

@tkriviradev
Copy link
Author

Sure, I will test this weekend and revert back, thank you!.

@defanator defanator transferred this issue from another repository Aug 19, 2022
@achawla2012
Copy link

As pointed out earlier by @nkashiv, @tkriviradev were you able to update your agent and check if that resolved issue?
Here are the instruction. https://amplify.nginx.com/docs/guide-installing-and-managing-nginx-amplify-agent.html#updating-the-agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@achawla2012 @tkriviradev @nkashiv