Minio console session error when exposed on subpath #2166

realies · 2022-07-11T07:34:56Z

Unable to expose the console on an nginx reverse proxy subpath using minio/minio:RELEASE.2022-07-08T00-05-23Z.

  minio:
    container_name: minio
    command: ["server", "/data", "--console-address", ":9001"]
    image: minio/minio:latest
    environment:
      - MINIO_SERVER_URL=http://localhost:9000
      - MINIO_BROWSER_REDIRECT_URL=http://localhost/console
    ports:
      - 9000:9000

		location /console/ {
			rewrite ^/console/(.*) /$1 break;
			proxy_pass http://minio:9001;
		}

Getting a 401 response on http://localhost/console/api/v1/session while everything else appears to load up fine:

{"code":401,"message":"unauthenticated for invalid credentials"}

The text was updated successfully, but these errors were encountered:

j13tw · 2022-07-11T09:11:46Z

#1908

I have the same problem
So, now I use RELEASE.2022-06-17T02-00-35Z (here )
and can touch to login to the website and view the console
However, the object download is creating an XMLRequest with a static path
So you can't download the object (it will hang....)

https://github.com/minio/console/blob/df38c84075f9926c381c0cbb1631ee51dbd468a9/portal-ui/src/screens/Console/Buckets/ListBuckets/Objects/utils.ts#L20-#L49

Also, when logging out
Its redirection is also #/subpath/subpath/login

After the RELEASE.2022-06-17T02-00-35Z Image Tag
Will got the same issue like your figure

djnnvx · 2022-07-16T16:18:02Z

I have attempted to use RELEASE.2022-06-17T02-00-35Z as well, but I cannot even login.
It seems that every time I try to login, I get the following error: "Expected element type <AssumeRoleResponse> but have <html>"

Is it maybe because i have my console setup on https://domain.com/s3-console and my bucket on https://domain.com/s3 ?

From my nginx logs:

nginx_1    | 2022-07-16T16:19:22.368763358Z 000.000.00.00 - - [16/Jul/2022:16:19:22 +0000] "POST /s3-admin/api/v1/login HTTP/1.0" 500 118 "https://domain.com/s3-admin/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36" "-"

What really confuses me, is that I can hit other endpoints just fine. I can get {"loginStrategy":"form"} from the /login endpoint, for instance.

mskyttner · 2022-07-19T15:23:43Z

I too am not able to expose minio console on subpath using latest version and also above mentioned versions (with nginx reverse proxy in front).

j13tw · 2022-07-20T00:56:01Z

I'm here to explain the practices and limits I set up for success
Currently using Kubernetes + Istio to build VS implementation
set ENV MINIO_BROWSER_REDIRECT_URL
Add "http://your.specify.domain/console"
"/console/" to specify console leads to "/" (port 9001)
The original MinIO S3 API uses "/" (port 9000)
In my solution, the console and MinIO s3 API are used same domain

Solve the problem that the Console download Object is abnormal
Need to refer to the #1908 settings

mskyttner · 2022-07-20T08:56:20Z

@j13tw Thanks for reaching out and it sounds like you confirm that there is a workaround available? I tried the following nginx configuration, which I think does what you suggest:

server {

	charset UTF-8;
	ignore_invalid_headers off;
	client_max_body_size 0;
	proxy_buffering off;

	location /minio-console {
            rewrite   ^/minio-console/(.*) /$1 break;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;

            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;

            proxy_connect_timeout 300;
            
            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            
            chunked_transfer_encoding off;
	    proxy_pass http://minio:9001;
        }


	location / {
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header Host $http_host;

		proxy_connect_timeout 300;
		# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
		proxy_http_version 1.1;
		proxy_set_header Connection "";
                chunked_transfer_encoding off;
		proxy_pass http://minio:9000;
	}

}

The docker-compose.yml has the MINIO_BROWSER_REDIRECT_URL set:

  minio:
    image: quay.io/minio/minio:latest
    volumes:
      - ./minio-share:/data/d1
    command: server --console-address=":9001" /data/d1
    environment:
      - MINIO_ROOT_USER=secret
      - MINIO_ROOT_PASSWORD=secret
      - MINIO_DOMAIN=data.bibliometrics.lib.kth.se
      - MINIO_SERVER_URL=https://data.bibliometrics.lib.kth.se
      - MINIO_BROWSER_REDIRECT_URL=https://data.bibliometrics.lib.kth.se/minio-console
#      - CONSOLE_SUBPATH=/minio-console
      - MINIO_API_SELECT_PARQUET=on
    healthcheck:
      test: ["CMD", "curl", "-f", "http://minio:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    restart: always

The result is that when I go to the domain where the minio server is running, I get a redirect to that domain with "/minio-console/" added, which stops due to a "401" as mentioned in the original issue here, since /minio-console/api/v1/session returns a JSON respone with code 401 and message "unauthenticated for invalid credentials":

The same "401" appears when using minio/minio:RELEASE.2022-06-17T02-00-35Z. With this version (everything else being the same) the redirect when hitting the minio server URL is "doubled" and instead goes to "/minio-console/minio-console/login". When manually fixing this and requesting "/minio-console/login" instead , I can see the login screen but not login with my credentials:

I think there is a main.js javascript POST happening to /minio-console/api/v1/login and the error message is {"code":500,"detailedMessage":"Post \"https://data.bibliometrics.lib.kth.se/\": dial tcp 127.0.1.1:443: connect: connection refused","message":"invalid Login"}.

This error mentions an IP and port which I'm quite sure I haven't configured anywhere...

I'm rolling back my attempt to expose minio console on a subpath using nginx reverse proxy in front, until I have a better understanding of how/if this is possible with or without a workaround.

Thankful for any more details/hints for reproducible workarounds.

hsinhoyeh · 2022-07-21T18:09:35Z

Is it maybe because i have my console setup on https://domain.com/s3-console and my bucket on https://domain.com/s3 ?

If you set the api under the path https://domain.com/s3, it could cause a routing issue to ingress when the console try to do STS login with WebIdentity, the request path starts /?Action=AssumeRoleWithWebIdentity... which nginx is unable forward traffic to the API server.

172.18.0.1 - - [21/Jul/2022:17:41:57 +0000] "POST /?Action=AssumeRoleWithWebIdentity&DurationSeconds=3600&Version=2011-06-15&WebIdentityToken=e<my-token> HTTP/1.1" 302 138 "-" "Go-http-client/1.1" 1064 0.000 [-] [] - - - - 1eefe30fd6a8f786fb9f8c54fa0bdd68

Here are logs found in my ingress, as you could see there is no destination ([-]) service shown (marked as bold).

add this environment variables could be helpful.

CONSOLE_MINIO_SERVER=https://domain.com/s3
which basically set STS endpoint.

code

func GetSTSEndpoint() string {
	return strings.TrimSpace(env.Get(ConsoleMinIOServer, "http://localhost:9000"))
}

harshavardhana · 2022-08-23T05:28:32Z

This actually works already in all situations

worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  4096;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;

    # include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server minio1:9000;
        server minio2:9000;
        server minio3:9000;
        server minio4:9000;
    }

    upstream console {
        ip_hash;
        server minio1:9001;
        server minio2:9001;
        server minio3:9001;
        server minio4:9001;
    }

    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;

        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }

        location /minio {
            rewrite   ^/minio/(.*) /$1 break;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;

            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;

            proxy_connect_timeout 300;
            
            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            
            chunked_transfer_encoding off;

            proxy_pass http://console;
        }
    }
}

version: '3.7'

# Settings and configurations that are common for all containers
x-minio-common: &minio-common
  image: quay.io/minio/minio:RELEASE.2022-08-13T21-54-44Z
  command: server --console-address ":9001" http://minio{1...4}/data{1...2}
  expose:
    - "9000"
    - "9001"
  environment:
    MINIO_ROOT_USER: minioadmin
    MINIO_ROOT_PASSWORD: minioadmin
    MINIO_BROWSER_REDIRECT_URL: http://10.0.0.185:9000/minio
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
    interval: 30s
    timeout: 20s
    retries: 3

# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
  minio1:
    <<: *minio-common
    hostname: minio1
    volumes:
      - data1-1:/data1
      - data1-2:/data2

  minio2:
    <<: *minio-common
    hostname: minio2
    volumes:
      - data2-1:/data1
      - data2-2:/data2

  minio3:
    <<: *minio-common
    hostname: minio3
    volumes:
      - data3-1:/data1
      - data3-2:/data2

  minio4:
    <<: *minio-common
    hostname: minio4
    volumes:
      - data4-1:/data1
      - data4-2:/data2

  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9001:9001"
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
  data1-1:
  data1-2:
  data2-1:
  data2-2:
  data3-1:
  data3-2:
  data4-1:
  data4-2:

So whatever is not working is perhaps generally a setup issue here. Also please use /minio instead of some custom names here. /minio is a reserved prefix that can be used here.

sudoexec · 2023-04-07T18:33:42Z

So whatever is not working is perhaps generally a setup issue here. Also please use /minio instead of some custom names here. /minio is a reserved prefix that can be used here.

I'm using RELEASE.2023-04-07T05-28-58Z.
You SHOULD NOT use /minio. Otherwise mc admin will not work. Because mc admin will access /minio/admin.

peterhirn · 2023-08-13T19:28:43Z

Doesn't work in Kubernetes with the nginx config posted by @harshavardhana.

Exposed locally as nodeport (30090 nodeport -> 8080 nginx -> 9001 console):

{
    "code": 401,
    "detailedMessage": "Post \"http://localhost:30090/\": dial tcp 127.0.0.1:30090: connect: connection refused",
    "message": "invalid Login"
}

Exposed via nginx-ingress on customer's cluster (custom PKI):

{
    "code": 401,
    "detailedMessage": "Post \"https://minio.customer.com/\": x509: certificate signed by unknown authority",
    "message": "invalid Login"
}

I need a option to tell the console where the minio instance is reachable (in this case simply http://minio:9000).

Update

Fixed by not setting MINIO_SERVER_URL (or setting it to http://localhost:9000). Confusing that the official documentation tells me to set it to the external fqdn.

ravindk89 · 2023-08-14T16:42:04Z

@peterhirn the referenced documentation is for Linux, where hosts would typically have some sort of externally-accessible FQDN.

You are on K8s, so there are an entirely different set of expectations around FQDN of pods/services.

We do not at this time intend to write detailed documentation on nginx reverse proxies in K8s. That's two steps removed from MinIO, and we don't have the resources to test/validate it against the myriad of possible K8s + DNS configurations out there.

We're discussing adding something of a disclaimer at the top guiding users to look at that reference as a baseline that may/will require modification, vs something guaranteed to work in a broad sense.

ocroquette · 2023-12-13T18:04:38Z

@peterhirn Thanks a lot for the hint about MINIO_SERVER_URL. It ended a long hour of uncertainty and frustration for me. I was setting up a MinIO server behind Caddy as a reverse proxy. Everything looked fine, but the login failed, as described above. I did set MINIO_SERVER_URL because it was recommended by the documentation and it made sense, but that was the issue. I removed it, and now everything works fine.

@ravindk89 I have full understanding that the documentation cannot cover all cases, but currently it says that MINIO_SERVER_URL maybe necessary if "The Console must use a specific hostname to connect or reference the MinIO Server, e.g. due to a reverse proxy or similar configuration.". This sounded like my case, and MINIO_SERVER_URL is mentioned in many examples online, so I set it, but it broke my setup. The troubleshooting is very hard then. Maybe the documentation could be more specific, or at least mention that this variable should not be set unless some specific problems occur.

ravindk89 · 2023-12-13T18:33:14Z

Maybe the documentation could be more specific, or at least mention that this variable should not be set unless some specific problems occur.

That's actually how we got to where we are now. Without that guidance, we had users who were frequently running into issues with the Console failing to load or redirecting incorrectly when reverse proxies were configured.

It would be easier if the errors were always the same for a given set of inputs, but that hasn't been my/our experience. It varies depending on MinIO version, K8s setup, nginx setup, and whether or not you're using Operators/custom charts/official charts/hand-written YAML.

I actually think this behavior overall might improve in newer versions of MinIO, as we are refining how the Console and Server talk to each other. I have to check how necessary this is now.

mmhlego · 2024-04-06T22:48:56Z

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }

        location /minio {
            rewrite   ^/minio/(.*) /$1 break;
            proxy_set_header Host $http_host;

@harshavardhana Thank you so much for your complete answer. Just a quick update:
If you want to access the minio api in /minio subpath and console UI in /console subpath, you should add the following line to the Nginx reverse proxy configuration:

	location /minio/ {
		rewrite   ^/minio/(.*) /$1 break;
		proxy_pass http://minio/;

		proxy_set_header Host $http_host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_connect_timeout 300;
		proxy_http_version 1.1;
		proxy_set_header Connection "";
		chunked_transfer_encoding off;
	}

        location /console {
		rewrite   ^/console/(.*) /$1 break;
		proxy_pass http://console/;

		proxy_http_version 1.1;
		proxy_set_header Host $http_host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-NginX-Proxy true;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
		real_ip_header X-Real-IP;
		proxy_connect_timeout 300;
		chunked_transfer_encoding off;
	}

mskyttner mentioned this issue Aug 18, 2022

feat: subpath support using reverse proxy #2174

Merged

harshavardhana closed this as completed Aug 23, 2022

harshavardhana added the working as intended label Aug 23, 2022

mskyttner mentioned this issue Apr 25, 2023

Console under subpath #2483

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minio console session error when exposed on subpath #2166

Minio console session error when exposed on subpath #2166

realies commented Jul 11, 2022

j13tw commented Jul 11, 2022 •

edited

Loading

djnnvx commented Jul 16, 2022 •

edited

Loading

mskyttner commented Jul 19, 2022

j13tw commented Jul 20, 2022

mskyttner commented Jul 20, 2022

hsinhoyeh commented Jul 21, 2022

harshavardhana commented Aug 23, 2022

sudoexec commented Apr 7, 2023

peterhirn commented Aug 13, 2023 •

edited

Loading

ravindk89 commented Aug 14, 2023 •

edited

Loading

ocroquette commented Dec 13, 2023

ravindk89 commented Dec 13, 2023

mmhlego commented Apr 6, 2024

Minio console session error when exposed on subpath #2166

Minio console session error when exposed on subpath #2166

Comments

realies commented Jul 11, 2022

j13tw commented Jul 11, 2022 • edited Loading

djnnvx commented Jul 16, 2022 • edited Loading

mskyttner commented Jul 19, 2022

j13tw commented Jul 20, 2022

mskyttner commented Jul 20, 2022

hsinhoyeh commented Jul 21, 2022

harshavardhana commented Aug 23, 2022

sudoexec commented Apr 7, 2023

peterhirn commented Aug 13, 2023 • edited Loading

ravindk89 commented Aug 14, 2023 • edited Loading

ocroquette commented Dec 13, 2023

ravindk89 commented Dec 13, 2023

mmhlego commented Apr 6, 2024

j13tw commented Jul 11, 2022 •

edited

Loading

djnnvx commented Jul 16, 2022 •

edited

Loading

peterhirn commented Aug 13, 2023 •

edited

Loading

ravindk89 commented Aug 14, 2023 •

edited

Loading