[SetupQuestion] nginxcontainererrors cannot connect to upstream

[warioishere]

Hello guys, this is not really a bug, but more a setup problem we have on our instance.
We run a public instance (https://invidious.yourdevice.ch) and with run it as a docker deployed instance with mutlipe containers restarting from time to time as suggested in the docs. We also added http3 proxy, and ipv6 log rotation to the setup.

When I check logs of the container nginx instance, (invidious-nginx-1 container) its full of those entries:

2024/06/03 07:57:55 [error] 29#29: *376409 connect() failed (111: Connection refused) while connecting to upstream, client: 172.24.0.1, server: , request: "GET /feed/popular HTTP/1.1", upstream: "http://[2001:db9::5]:3000/feed/popular", host: "invidious.yourdevice.ch", referrer: "https://invidious.yourdevice.ch/"

An I mean really full. We dont have problems on the instance. Videos do load fast, everything plays fast. No problem at all. Still those logs bother me a bit. Seems like the nginx container cant reach the invidious containers? But if it would be so, then the server wouldnt work at all? Can you guys gimme a hint?

@unixfox @bugmaschine @perennialtech ?

This is our setup:

Nginx Reverse Proxy:

server {
    listen 80;
    listen [::]:80;
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name invidious.yourdevice.ch;

    access_log off;
    error_log /var/log/nginx/error.log crit;

    ssl_certificate /etc/letsencrypt/live/invidious.yourdevice.ch/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/invidious.yourdevice.ch/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;    # so Invidious knows domain
        proxy_http_version 1.1;     # to keep alive
        proxy_set_header Connection ""; # to keep alive
    }

    location ~ (^/videoplayback|^/vi/|^/ggpht/|^/sb/) {
        proxy_buffering on;
        proxy_buffers 1024 16k;
        proxy_set_header X-Forwarded-For "";
        proxy_set_header CF-Connecting-IP "";
        proxy_hide_header "alt-svc";
        sendfile on;
        sendfile_max_chunk 512k;
        tcp_nopush on;
        aio threads=default;
        aio_write on;
        directio 16m;
        proxy_hide_header Cache-Control;
        proxy_hide_header etag;
        proxy_http_version 1.1;
        proxy_set_header Connection keep-alive;
        proxy_max_temp_file_size 32m;
        access_log off;
        proxy_pass http://unix:/opt/http3-ytproxy/http-proxy.sock;
        add_header Cache-Control private always;
}

    if ($https = '') { return 301 https://$host$request_uri; }  # if not connected to HTTPS, perma-redirect to HTTPS
}

This is the nginx setup for the invidious-nginx container:

user www-data;
events {
    worker_connections 1000;
}
http {
    server {
        listen 3000;
        listen [::]:3000;
        access_log off;

        location / {
            resolver 127.0.0.11;
            set $backend "invidious";
            proxy_pass http://$backend:3000;
            proxy_http_version 1.1; # to keep alive
            proxy_set_header Connection ""; # to keep alive
        }
    }
}

This is our docker-compose.yml

version: "3"
services:
    invidious:
        image: quay.io/invidious/invidious:latest
        deploy:
            replicas: 6
        restart: unless-stopped
        environment:
            INVIDIOUS_CONFIG: |
                channel_threads: 0
                feed_threads: 0
                db:
                    dbname: invidious
                    user: kemal
                    password: kemal
                    host: invidious-db
                    port: 5432
                check_tables: true
                external_port: 443
                domain: invidious.yourdevice.ch
                https_only: true
                statistics_enabled: true
                force_resolve: ipv6
                hmac_key: "xxx"
                #  banner: "by yourdevice.ch"
                #  popular_enabled: true
                registration_enabled: true
                login_enabled: true
                captcha_enabled: true
                enable_user_notifications: true
                use_pubsub_feeds: true
                use_innertube_for_captions: true
                jobs:
                  clear_expired_items:
                    enabled: false
                  refresh_channels:
                    enabled: false
                  refresh_feeds:
                    enabled: false
        healthcheck:
            test: wget -nv --tries=1 --spider http://127.0.0.1:3000/api/v1/stats || exit 1
            interval: 30s
            timeout: 5s
            retries: 2
        logging:
            options:
                max-size: "1G"
                max-file: "4"
        depends_on:
           - invidious-db

    invidious-refresh:
        image: quay.io/invidious/invidious:latest
        restart: unless-stopped
        environment:
            INVIDIOUS_CONFIG: |
                db:
                    dbname: invidious
                    user: kemal
                    password: kemal
                    host: invidious-db
                    port: 5432
                    check_tables: true
                check_tables: true
                external_port: 443
                domain: invidious.yourdevice.ch
                https_only: true
                statistics_enabled: true
                force_resolve: ipv6
                hmac_key: "xxx"
                #  banner: "by yourdevice.ch"
                #  popular_enabled: true
                registration_enabled: true
                login_enabled: true
                captcha_enabled: true
                enable_user_notifications: true
                use_pubsub_feeds: true
                use_innertube_for_captions: true
                jobs:
                  clear_expired_items:
                    enabled: false
                  refresh_channels:
                    enabled: false
                  refresh_feeds:
                    enabled: false
        healthcheck:
            test: wget -nv --tries=1 --spider http://127.0.0.1:3000/api/v1/stats || exit 1
            interval: 30s
            timeout: 5s
            retries: 2
        logging:
            options:
                max-size: "1G"
                max-file: "4"
        depends_on:
           - invidious-db

    nginx:
        image: nginx:latest
        restart: unless-stopped
        volumes:
            - ./nginx.conf/nginx.conf:/etc/nginx/nginx.conf:ro
        depends_on:
            - invidious
        ports:
            - "3000:3000"

    http3-ytproxy:
        image: 1337kavin/ytproxy:latest
        restart: unless-stopped
        user: "33:33"
        network_mode: "host"
        environment:
            DISABLE_WEBP: 1
        volumes:
           - /opt/http3-ytproxy:/app/socket

    invidious-db:
        image: docker.io/library/postgres:14
        restart: unless-stopped
        volumes:
          - postgresdata:/var/lib/postgresql/data
          - ./config/sql:/config/sql
          - ./docker/init-invidious-db.sh:/docker-entrypoint-initdb.d/init-invidious-db.sh
        environment:
            POSTGRES_DB: invidious
            POSTGRES_USER: kemal
            POSTGRES_PASSWORD: kemal
        healthcheck:
            test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]

volumes:
    postgresdata:

networks:
  default:
    enable_ipv6: true
    ipam:
      config:
        - subnet: 2001:0DB9::/112
          gateway: 2001:0DB9::1

• we have set ipv6 enabled in etc/docker/daemon.json
• we restart docker containers as suggested
• we run ipv6rotation twice a day
• we have changed the owner of /opt/http3-ytproxy to www-data:www-data

Thanks for having a look!

Cheers guys

[perennialtech]

@warioishere For this question, you may have better luck asking in the Invidious Instance Owners room on Matrix (see #521 (comment)).

Also, since your hmac_key is still visible in the edit history of your issue and in emails sent to people watching this repository, you should generate a new key to replace it completely.

[warioishere]

key is already regenerated because I realized it when reading through again. If anyone knows about this issue, let me know.

[unixfox]

I think it's probably because nginx is trying to send the traffic in ipv6 to the container, but the container does not listen on ipv6.

There is an issue about it here: iv-org/invidious#4705

One temporary way would be to force Invidious to listen on IPv6 by adding this parameter in the invidious config:

host_binding: [::]

[warioishere]

host_binding: [::]

add this to invidious and invidious-refresh config?

[unixfox]

host_binding: [::]

add this to invidious and invidious-refresh config?

Only the docker containers that receive traffic from NGINX.
So invidious only.

[warioishere]

Okay thanks, will try this evening as soon as I come home. The big question to me is, why does the setup still work if the containers cannot receive traffic from nginx

[unixfox]

Okay thanks, will try this evening as soon as I come home. The big question to me is, why does the setup still work if the containers cannot receive traffic from nginx

It's called happy eyeballs: https://en.wikipedia.org/wiki/Happy_Eyeballs

If the software can't connect in ipv6, it will try in ipv4 after.

[warioishere]

Okay thanks, will try this evening as soon as I come home. The big question to me is, why does the setup still work if the containers cannot receive traffic from nginx

It's called happy eyeballs: https://en.wikipedia.org/wiki/Happy_Eyeballs

If the software can't connect in ipv6, it will try in ipv4 after.

Okay thanks, understood

adding host_binding: [::] to the invidious container is making the instance not work at all anymore. I get the same logs as before, and additional to that refused connection on container IPv4 adresses

[unixfox]

what are the invidious logs though?

[warioishere]

what are the invidious logs though?

the nginx container logs are the same plus:

2024/06/03 07:57:55 [error] 29#29: *376409 connect() failed (111: Connection refused) while connecting to upstream, client: 172.24.0.1, server: , request: "GET /feed/popular HTTP/1.1", upstream: "http://**containeripv4adresses**:3000/feed/popular", host: "invidious.yourdevice.ch", referrer: "https://invidious.yourdevice.ch/"

didnt check the invidious container logs.
Just no more access at all to invidious web (nginx bad gateway)

do you need the invidious container logs? I reverted instantly because running this as a public instance.

[unixfox]

yes try to see if invidious process is starting normally in order to know what's really going on.