Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make check_port an async function #4677

Merged
merged 11 commits into from
Dec 5, 2023

Conversation

agners
Copy link
Member

@agners agners commented Nov 7, 2023

Proposed change

This makes check_port() an async function. This also requires to change the ingress_port property to a async method.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New feature (which adds functionality to the supervisor)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:
  • Link to cli pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • The code has been formatted using Black (black --fast supervisor tests)
  • Tests have been added to verify that the new code works.

If API endpoints of add-on configuration are added/changed:

@agners agners added the refactor A code change that neither fixes a bug nor adds a feature label Nov 7, 2023
@agners agners changed the title Make check_port asyncio Make check_port an async function Nov 7, 2023
@bdraco
Copy link
Member

bdraco commented Nov 8, 2023

Tweaked it a bit #4678

@agners agners force-pushed the use-asyncio-check-port branch from 192d473 to 79d10dd Compare November 8, 2023 11:05
@agners
Copy link
Member Author

agners commented Nov 8, 2023

I am not very happy how I have to handle the ingress_port port property here. The property currently implicitly gathers a port. When that exactly happens is really depending on when the Add-on info API is being used or the ingress URL is being generated. This code also did I/O (socket connection) in the asyncio event loop, which is not ideal.

The current change converts the property to a async get method which takes the I/O out of the asyncio event loop. But when the port is exactly determinated is still very implicit.

There is a tradeoff of reserving ports unnecessarily early and doing port reservation as late as possible. Ideally I think the port should be determined on first add-on startup. I don't think we need to clear the port on stop.

But this would mean that the URL will not be valid until the add-on gets started, not sure if we handle that properly.

In any case, this can be done in a follow-up PR I think.

@bdraco
Copy link
Member

bdraco commented Nov 8, 2023

I am not very happy how I have to handle the ingress_port port property here. The property currently implicitly gathers a port. When that exactly happens is really depending on when the Add-on info API is being used or the ingress URL is being generated. This code also did I/O (socket connection) in the asyncio event loop, which is not ideal.

Its definitely not ideal. The property having the side effect of allocating a port was a bit scarier to me since it was non-obvious what was going on under the hood.

The current change converts the property to a async get method which takes the I/O out of the asyncio event loop. But when the port is exactly determinated is still very implicit.

It seems like it should be refactored into a PortManager class or something similar that knows about all the ports

There is a tradeoff of reserving ports unnecessarily early and doing port reservation as late as possible. Ideally I think the port should be determined on first add-on startup. I don't think we need to clear the port on stop.

That makes more sense to me, I don't think there is a risk we run out of ports.

I think this is a good change but a followup to refactor the ingress_port area would make me feel a lot better about it

@bdraco
Copy link
Member

bdraco commented Nov 8, 2023

I'll throw this on my production now to make sure nothing breaks

@bdraco
Copy link
Member

bdraco commented Nov 8, 2023

Nov 08 14:47:40 homeassistant hassio_supervisor[540]: 23-11-08 08:47:40 ERROR (MainThread) [aiohttp.server] Error handling request
Nov 08 14:47:40 homeassistant hassio_supervisor[540]: Traceback (most recent call last):
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     resp = await request_handler(request)
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/aiohttp.py", line 139, in sentry_app_handle
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     reraise(*_capture_exception(hub))
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/sentry_sdk/_compat.py", line 115, in reraise
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     raise value
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/aiohttp.py", line 129, in sentry_app_handle
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     response = await old_handle(self, request)
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 543, in _handle
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     resp = await handler(request)
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 114, in impl
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     return await handler(request)
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 186, in block_bad_requests
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     return await handler(request)
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 202, in system_validation
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     return await handler(request)
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 221, in token_validation
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     return await handler(request)
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 280, in core_proxy
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     return await handler(request)
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/ingress.py", line 141, in handler
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     return await self._handle_request(request, addon, path, session_data)
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/ingress.py", line 217, in _handle_request
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     async with self.sys_websession.request(
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1187, in __aenter__
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     self._resp = await self._coro
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:                  ^^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 445, in _request
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     url = self._build_url(str_or_url)
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 372, in _build_url
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     url = URL(str_or_url)
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:           ^^^^^^^^^^^^^^^
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/yarl/_url.py", line 179, in __new__
Nov 08 14:47:40 homeassistant hassio_supervisor[540]:     raise TypeError("Constructor parameter should be str")
Nov 08 14:47:40 homeassistant hassio_supervisor[540]: TypeError: Constructor parameter should be str
Nov 08 14:47:41 homeassistant hassio_supervisor[540]: 23-11-08 08:47:41 ERROR (MainThread) [aiohttp.server] Error handling request
Nov 08 14:47:41 homeassistant hassio_supervisor[540]: Traceback (most recent call last):
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     resp = await request_handler(request)
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/aiohttp.py", line 139, in sentry_app_handle
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     reraise(*_capture_exception(hub))
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/sentry_sdk/_compat.py", line 115, in reraise
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     raise value
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/aiohttp.py", line 129, in sentry_app_handle
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     response = await old_handle(self, request)
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 543, in _handle
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     resp = await handler(request)
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 114, in impl
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     return await handler(request)
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 186, in block_bad_requests
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     return await handler(request)
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 202, in system_validation
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     return await handler(request)
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 221, in token_validation
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     return await handler(request)
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 280, in core_proxy
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     return await handler(request)
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/ingress.py", line 141, in handler
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     return await self._handle_request(request, addon, path, session_data)
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/src/supervisor/supervisor/api/ingress.py", line 217, in _handle_request
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     async with self.sys_websession.request(
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1187, in __aenter__
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     self._resp = await self._coro
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:                  ^^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 445, in _request
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     url = self._build_url(str_or_url)
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 372, in _build_url
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     url = URL(str_or_url)
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:           ^^^^^^^^^^^^^^^
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:   File "/usr/local/lib/python3.11/site-packages/yarl/_url.py", line 179, in __new__
Nov 08 14:47:41 homeassistant hassio_supervisor[540]:     raise TypeError("Constructor parameter should be str")
Nov 08 14:47:41 homeassistant hassio_supervisor[540]: TypeError: Constructor parameter should be str

Looks like there might be an issue with the url construction

@bdraco
Copy link
Member

bdraco commented Nov 8, 2023

/usr/local/lib/python3.11/site-packages/awesomeversion/awesomeversion.py:455: RuntimeWarning: coroutine 'APIIngress._create_url' was never awaited

Copy link
Member

@bdraco bdraco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like there is a missing await

diff --git a/supervisor/api/ingress.py b/supervisor/api/ingress.py
index 75179361..1817071a 100644
--- a/supervisor/api/ingress.py
+++ b/supervisor/api/ingress.py
@@ -201,7 +201,7 @@ class APIIngress(CoreSysAttributes):
         session_data: IngressSessionData | None,
     ) -> web.Response | web.StreamResponse:
         """Ingress route for request."""
-        url = self._create_url(addon, path)
+        url = await self._create_url(addon, path)
         source_header = _init_header(request, addon, session_data)
 
         # Passing the raw stream breaks requests for some webservers

@home-assistant home-assistant bot marked this pull request as draft November 8, 2023 14:50
@home-assistant
Copy link

home-assistant bot commented Nov 8, 2023

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

@bdraco
Copy link
Member

bdraco commented Nov 10, 2023

testing this on my production now

Copy link
Member

@bdraco bdraco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on production, all good now 👍

@bdraco
Copy link
Member

bdraco commented Nov 10, 2023

Screenshot 2023-11-10 at 10 05 33 AM

@agners agners marked this pull request as ready for review November 10, 2023 17:19
@home-assistant home-assistant bot requested a review from bdraco November 10, 2023 17:19
@bdraco
Copy link
Member

bdraco commented Nov 13, 2023

strace of check_port

socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 3
ioctl(3, FIONBIO, [1])                  = 0
connect(3, {sa_family=AF_INET, sin_port=htons(8123), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation in progress)
poll([{fd=3, events=POLLOUT|POLLERR}], 1, 500) = 1 ([{fd=3, revents=POLLOUT}])
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
close(3)                                = 0

strace of async_check_port

socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 6
ioctl(6, FIONBIO, [1])                  = 0
connect(6, {sa_family=AF_INET, sin_port=htons(8123), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation in progress)
epoll_ctl(3, EPOLL_CTL_ADD, 6, {events=EPOLLOUT, data={u32=6, u64=140595754434566}}) = 0
epoll_pwait(3, [{events=EPOLLOUT, data={u32=6, u64=140595754434566}}], 2, 500, NULL, 8) = 1
getsockopt(6, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
epoll_pwait(3, [{events=EPOLLOUT, data={u32=6, u64=140595754434566}}], 2, 0, NULL, 8) = 1
epoll_ctl(3, EPOLL_CTL_DEL, 6, 0x7ffc0d5cfedc) = 0
close(6)                                = 0

@bdraco
Copy link
Member

bdraco commented Nov 13, 2023

Both do a 0.5s poll, the async version also does a 0s poll (returns right away as well) and has to add/remove the epoll, but the async version doesn't have the executor overhead

@bdraco
Copy link
Member

bdraco commented Nov 13, 2023

Copy link
Member

@pvizeli pvizeli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx @bdraco for the callstack

@home-assistant home-assistant bot marked this pull request as draft November 13, 2023 17:18
@agners agners requested a review from pvizeli November 16, 2023 14:25
@agners agners marked this pull request as ready for review November 16, 2023 14:34
@home-assistant home-assistant bot requested a review from bdraco November 16, 2023 14:34
@agners
Copy link
Member Author

agners commented Nov 16, 2023

I guess another question is why do the check here https://github.com/home-assistant/supervisor/pull/4677/files#diff-063212876158cc6d68b7582cea3335ca047c893162509e7f1ee1a1bfa89abf2eR143

Maybe it would make sense to do the api calls with a shorter https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientTimeout ?

I guess the intention of the code was to fail faster when the port is still closed? But I'd expect that aiohttp returns quickly in that case too no?

I do agree I think we should remove the port check in that case.

@agners
Copy link
Member Author

agners commented Nov 16, 2023

Could we maybe do that change in a separate PR? I am a bit scared that the change will subtle change the behavior of check_api_state() which is used in quite some places 🤔

@home-assistant home-assistant bot marked this pull request as draft November 16, 2023 22:14
@agners agners marked this pull request as ready for review November 21, 2023 15:42
@home-assistant home-assistant bot requested a review from mdegat01 November 21, 2023 15:42
@agners agners requested a review from pvizeli November 22, 2023 22:44
@mdegat01 mdegat01 force-pushed the use-asyncio-check-port branch from dde9b06 to 5301baf Compare December 4, 2023 21:59
@mdegat01 mdegat01 merged commit 883e54f into home-assistant:main Dec 5, 2023
@agners agners mentioned this pull request Dec 6, 2023
12 tasks
@github-actions github-actions bot locked and limited conversation to collaborators Dec 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla-signed refactor A code change that neither fixes a bug nor adds a feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants