help request: Chrome keeps port :9443 after http_to_https. Despite explicit pluginAttrs configuration for 443

[reagan-mcfadden]

Description

Hello, I am experiencing an issue with the redirect plugin, http_to_https and chrome. The result is that :9443 is displayed in the url after redirect on chrome. Other browsers (firefox, brave, arc) instead properly route traffic to 443 and do not append :9443. Yes, I have tried clearing cache and running from different clients.

Edit a bit more info. 😠
This seems to be having different outcomes on different user clients. On mac clients All browsers appear to be reporting :9443, however windows and linux clients firefox and select browsers don't include the :9443

I've looked through a number of other issues like: #7011, #7172, #4942 and have found either outdated or invalid configuration options that do not resolve my issue. The https://apisix.apache.org/docs/apisix/plugins/redirect/ has a blurb in the note about the priority of the port used in http_to_https. According to it I have configured the correct first priority value.

Otherwise https behaves in an expected way. Below you can find enough information to reproduce the environment.

My apisix.yaml contains the following rule:

services:
  - name: Website
    routes:
      - uris:
          - /*
        name: web-entry
        plugins:
          redirect:
            http_to_https: true
    upstream:
      type: roundrobin
      nodes:
        - host: 1xx.xxx.xxx.xxx
          port: 443
          weight: 1
       - host: 2xx.xxx.xxx.xxx
          port: 443
          weight: 1
       - host: 3xx.xxx.xxx.xxx
          port: 443
          weight: 1
      pass_host: pass
      upstream_host: xxx.com
      scheme: https

My config.yaml has the following pluginAttrs values.

plugin_attr:
  redirect: #force redirect so its not exit port.
    https_port: 443

My docker-compose that is used for the server environment:

services:
  apisix:
    build:
      context: ./
      dockerfile: Dockerfile
    restart: always
    volumes:
      - ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
      - ./apisix_conf/apisix.yaml:/usr/local/apisix/conf/apisix.yaml:ro
      - ./.env:/usr/local/apisix/.env
      - /etc/letsencrypt/live/:/etc/letsencrypt/live/:ro
      - /etc/letsencrypt/archive/:/etc/letsencrypt/archive/:ro
    depends_on:
      - etcd
    ##network_mode: host
    ports:
      - "9180:9180/tcp"
      - "80:9080/tcp"
      - "9091:9091/tcp"
      - "443:9443/tcp"
      - "9443:9443/tcp"
      - "9092:9092/tcp"
    networks:
      apisix:

  etcd:
    image: bitnami/etcd:3.5.11
    restart: always
    volumes:
      - etcd_data:/bitnami/etcd
    environment:
      ETCD_ENABLE_V2: "true"
      ALLOW_NONE_AUTHENTICATION: "yes"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd:2379"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
    ports:
      - "2379:2379/tcp"
    networks:
      apisix:

networks:
  apisix:
    driver: bridge

volumes:
  etcd_data:
    driver: local

Modified dockerfile to include adc for deployments.

# Docker file for apisix enchanced with api/adc
FROM apache/apisix:3.10.0-debian

# Install curl
USER root
RUN apt-get update
RUN apt-get install -y curl

USER root
# Install adc https://github.com/api7/adc/
RUN curl -sL "https://run.api7.ai/adc/install" | sh -x && \
   which adc && \
   ls -l $(which adc) && \
   echo $PATH

Relevant CI/CD deployment (bitbucket) instructions:

definitions:
  steps:
    - step: &deploy
        image: alpine:3.20.0
        name: Deploy
        script:
          - apt-get update
          - apt-get install -qq unzip openssh-client rsync
          - env | while IFS='=' read -r n v; do echo  "$n=$v" >> .env; done
          - rsync -aiRv --stats --progress --exclude-from=.rsync/excludes.txt * $USERNAME@$PUBLIC_IP:$DEPLOY_DIR
          - ssh $USERNAME@$PUBLIC_IP "cd $DEPLOY_DIR && bash scripts/ssl-cert.sh ${DOMAIN}"
          - ssh $USERNAME@$PUBLIC_IP "cd $DEPLOY_DIR && docker compose exec apisix adc sync -f conf/apisix.yaml"

.env contains something roughly like:

ADC_SERVER=http://0.0.0.0:9180
ADC_TOKEN=XXXXXX
DOMAIN=xxx.env

scripts/ssl-cert.sh

#!/bin/bash

if [ -z "$1" ]; then
  echo "Usage: $0 <domain>"
  exit 1
fi

DOMAIN=$1

CERT_FILE="/etc/letsencrypt/live/$DOMAIN/fullchain.pem"
KEY_FILE="/etc/letsencrypt/live/$DOMAIN/privkey.pem"
CONFIG_FILE="conf/apisix.yaml"


CERT_CONTENT=$(cat $CERT_FILE)
KEY_CONTENT=$(cat $KEY_FILE)

cp "$CONFIG_FILE" "${CONFIG_FILE}.bak"

{
  echo ""
  echo "ssls:"
  echo "  -"
  echo "    snis:"
  echo "      - $DOMAIN"
  echo "    certificates:"
  echo "      - certificate: |"
  printf '%s\n' "$CERT_CONTENT" | sed 's/^/          /'  # Adjusted for extra indentation
  echo "        key: |"
  printf '%s\n' "$KEY_CONTENT" | sed 's/^/          /'  # Adjusted for extra indentation
} >> "$CONFIG_FILE"

echo "Updated $CONFIG_FILE with new SSL configuration for $DOMAIN"

Any pointers or suggestion to resolve chrome redirecting to https://xxx.com:9443 from http://xxx.com would be welcome.

Environment

• Apisix version: 3.9.1
• Operating system: Linux a413f91e0851 6.8.0-31-generic optimize: used lua table to cache the ngx variable. #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 GNU/Linux
• OpenResty / Nginix version: openresty/1.25.3.1
• etcd version: 3.5.11

Note this is being deployed as a container on a ubuntu digital ocean droplet. Droplet has the following configuration/installation for newest ubuntu version.

sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install certbot
apt-cache policy docker-ce
sudo apt install docker-ce

ufw: 0.36.2

sudo ufw enable
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 9180/tcp #in memory config
sudo ufw allow 9443/tcp

cert:

sudo certbot certonly --standalone -d xxx.com

[reagan-mcfadden]

To add to above. It is really inconsistent which clients get the redirect with and without ports it seems to be entirely related to dom.security.https_first_pbm config in chromium based products. If the value is set to false I get the undesired behaviour. Which leads me to believe the override is in pluginAttrs is effectively not doing anything.

I also noticed that 9.10.0 is out so I upgraded to it with no change in outcomes.

[reagan-mcfadden]

The issue seems to be client side 301 caches. Having clients clear their caches resolved the issue.