[BUG] frequent 504 behind NPM

[working-name]

Describe the bug
For some reason I find myself doing docker restart the more I use search throughout the day. The container is just sitting pretty there, no load. Nothing in NPM's logs other than timeout connecting to whoogle container. Private instance, I see nothing in NPM logs to make me think someone anyone else is hitting it outside of me.

2024/06/26 13:24:21 [error] 878044#878044: *597529 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 1.1.1.1, server: whoogle.domain, request: "GET /search?q=Lumen HTTP/2.0", upstream: "http://172.17.0.1:18084/search?q=Lumen", host: "whoogle.domain"

Container sitting pretty, not choked, all the while ignoring incoming requests. Normally PIDS are about 8 when idle, so there's a few hanging for sure.

CONTAINER ID   NAME             CPU %     MEM USAGE / LIMIT   MEM %     NET I/O        BLOCK I/O       PIDS
4be21cef5885   whoogle-search   0.15%     128.6MiB / 1GiB     12.56%    102MB / 75MB   108MB / 168kB   15

To Reproduce
Steps to reproduce the behavior:

1. Run whoogle via docker, place it behind NPM
2. Use as you would normally
3. Bump into 504 errors

Deployment Method

• Heroku (one-click deploy)
• Docker
• run executable
• pip/pipx
• Other: [describe setup]

Version of Whoogle Search

• Latest build from [Docker Hub, benbusby/whoogle-search]
• Version [Whoogle Search v0.8.4]

Desktop (please complete the following information):

• OS: [Debian GNU/Linux 12 (bookworm)]
• Browser [any]
• Version [any]

Smartphone (please complete the following information): NOT applicable.

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
The container either recovers by itself after a while (is it single threaded and it waits for something to time out?), or it never does and I have to manually restart it.

It's not NPM, I checked logs, it passes the request along. I SSHed into the box, and ran a curl against whoogle container's locally exposed port, and it times out. It's 100% whoogle.

There's no useful errors logs in docker logs except this repeating issue with simple-theme.css:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1348, in handle_user_exception
    return self.ensure_sync(handler)(e)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/whoogle/app/routes.py", line 624, in internal_error
    return render_template(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/templating.py", line 151, in render_template
    return _render(app, template, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/templating.py", line 132, in _render
    rv = template.render(context)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/whoogle/app/templates/error.html", line 8, in top-level template code
    <link rel="stylesheet" href="{{ cb_url(config.theme + '-theme.css') }}"/>
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/whoogle/app/__init__.py", line 179, in <lambda>
    cb_url=lambda f: app.config['CACHE_BUSTING_MAP'][f])
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
KeyError: 'simple-theme.css'

[working-name]

Found it dead again. This time the container says unhealthy. I just issued a killall -9 curl to kill all the health check processes that hung. Here's what's running on it:

$ docker compose exec whoogle-search ps aux
PID   USER     TIME  COMMAND
    1 whoogle   0:00 {run} /bin/sh ./run
    7 whoogle   0:00 {start-tor.sh} /bin/sh misc/tor/start-tor.sh
   11 whoogle   0:12 tor -f /etc/tor/torrc
   15 whoogle   0:59 python3 -um app --host 0.0.0.0 --port 5000
12776 whoogle   0:00 ps aux

Trying to curl from within the container results in the same behavior: timeout. Does this have to do with Tor? I don't remember enabling it.

$ docker compose exec whoogle-search curl localhost:5000 -vvv -m 2
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
* Operation timed out after 2002 milliseconds with 0 bytes received
* Closing connection
curl: (28) Operation timed out after 2002 milliseconds with 0 bytes received

Okay, let's rebuild the container.

$ docker compose exec whoogle-search curl localhost:5000 -vvv -m 2; docker compose up -d --force-recreate && sleep 10 && docker compose exec whoogle-search curl localhost:5000 -vvv -m 2
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
* Operation timed out after 2002 milliseconds with 0 bytes received
* Closing connection
curl: (28) Operation timed out after 2002 milliseconds with 0 bytes received
[+] Running 1/1
 ✔ Container whoogle-search  Started                                                                                                                                            20.2s
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
* Operation timed out after 2002 milliseconds with 0 bytes received
* Closing connection
curl: (28) Operation timed out after 2002 milliseconds with 0 bytes received
### 10 seconds was not enough for the process to be ready...
containme@geek23:~/apps/whoogle-search$ docker compose exec whoogle-search curl localhost:5000 -vvv -m 2
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: max-age=86400
< Content-Length: 12765
< Content-Type: text/html; charset=utf-8
< Date: Thu, 27 Jun 2024 03:21:07 GMT
< Server: waitress
< Set-Cookie: __Secure-session=.eJw1zMsKgkAUANBfibt2wEc-ElrmokDQRoU2cZ1H2ugU5kAl_nuzaXdWZ4HrU0wjaqFnSOfJCAfQzN3f7KFlf4N0WR1Q4mMBmxZSyN2waLMBc3_omrv6nvXwLrxOtqouK3rpyoDneKhPdcPCagxfNDhONMtURd092MuYntuGyZh7AhOCPjKy3blIEtm6JGERjzGIrD1Yf9UWNKw.Znzaow.JVGZZBm2hG4RsYpAPSdyjLnUzgU; Expires=Fri, 27 Jun 2025 03:21:07 GMT; Secure; HttpOnly; Path=/; SameSite=Lax
< Vary: Cookie
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
<
<html style="background: #000;">
<head>
... and so on. Normal response.