[installer] minio pod on k3s in crashloop

[SirLemyDanger]

Bug description

Minio pod keeps crashing due to a failed liveliness probe.

Steps to reproduce

1. Install k3s (without traefik) on a single node
2. install gitpod with installer (version see below in quote)
3. minio pod keeps crashing

./installer version
{
"version": "main.1982",
"components": {
"agentSmith": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
},
"blobserve": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
},
"caUpdater": {
"version": "commit-fc0c0a613c354444056ff43dfcc55c5acc9052a6"
},
"contentService": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
},
"dashboard": {
"version": "commit-c0b7a94a6c554f02916225320c1679297117e9dd"
},
"dbMigrations": {
"version": "commit-8bdc71a55d1e6a9e265a160c96d099f7708a43ae"
},
"dbSync": {
"version": "commit-8bdc71a55d1e6a9e265a160c96d099f7708a43ae"
},
"imageBuilder": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
},
"imageBuilderMk3": {
"version": "commit-5837426c7eeddeddfe8758ffff947e77da8501d2",
"builderImage": {
"version": "505263dc3590c6d680bf464ccba85b1bac5d444d"
}
},
"integrationTests": {
"version": ""
},
"kedge": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
},
"openVSXProxy": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
},
"paymentEndpoint": {
"version": "commit-8bdc71a55d1e6a9e265a160c96d099f7708a43ae"
},
"proxy": {
"version": "commit-6ab6ab2a782a009a5827a7bc1c5c59d96a192b87"
},
"registryFacade": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
},
"server": {
"version": "commit-8bdc71a55d1e6a9e265a160c96d099f7708a43ae"
},
"serviceWaiter": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
},
"workspace": {
"codeImage": {
"version": "commit-b1a73b372191b23ddec0a56c7f00d3ce59712d16"
},
"dockerUp": {
"version": "commit-b1a73b372191b23ddec0a56c7f00d3ce59712d16"
},
"supervisor": {
"version": "commit-8bdc71a55d1e6a9e265a160c96d099f7708a43ae"
},
"desktopIdeImages": {
"codeDesktop": {
"version": "commit-1ed5952d58e747a3fab45211b742a30469e26966"
},
"codeDesktopInsiders": {
"version": "commit-1ed5952d58e747a3fab45211b742a30469e26966"
},
"intellij": {
"version": "commit-b6c0295c4a7191d9b9cdfd0f99123c2d1b9cb12a"
},
"goland": {
"version": "commit-a6f925ad0754a1649412b338fba757c98cb76f72"
},
"pycharm": {
"version": "commit-64539bf61cda420e49e638422ac1afef1c319b4a"
},
"phpstorm": {
"version": "commit-74d2f69cf729eed4f5a95039a2802c43c0dac2c1"
}
}
},
"wsDaemon": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2",
"userNamespaces": {
"seccompProfileInstaller": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
},
"shiftfsModuleLoader": {
"version": "commit-b1a73b372191b23ddec0a56c7f00d3ce59712d16"
}
}
},
"wsManager": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
},
"wsManagerBridge": {
"version": "commit-8bdc71a55d1e6a9e265a160c96d099f7708a43ae"
},
"wsProxy": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
},
"wsScheduler": {
"version": "commit-1c0c677313692945976566d0d042e4f033889de2"
}
}
}

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[SirLemyDanger]

describe_minio.txt
logs_minio.txt

[sagor999]

Got this same issue when starting up cluster in workspace-preview.
Turning on debug more produced this:

API: SYSTEM()                                                                                                                                                                                       
Time: 23:16:49 UTC 01/19/2022                                                                                                                                                                       
DeploymentID: 22e221ca-240b-46ef-9386-12d45b895816                                                                                                                                                  
Error: Access Denied. (*errors.errorString)                                                                                                                                                         
       requestHeaders={"method":"GET","reqURI":"/minio/admin/v3/info","header":{"Host":["localhost:9000"],"User-Agent":["MinIO (linux; amd64) madmin-go/0.0.1 mc/DEVELOPMENT.2021-10-07T06-12-04Z"],
       4: cmd/auth-handler.go:154:cmd.validateAdminSignature()                                                                                                                                      
       3: cmd/auth-handler.go:165:cmd.checkAdminRequestAuth()                                                                                                                                       
       2: cmd/admin-handlers.go:1977:cmd.adminAPIHandlers.ServerInfoHandler()                                                                                                                       
       1: net/http/server.go:2046:http.HandlerFunc.ServeHTTP()                                                                                                                                      
minio 23:16:49.43 INFO  ==> MinIO is already stopped...                                                                                                                                             

My existing cluster got messed up before I was able to track this issue down, but might be related to this:
bitnami/charts#5951

Will update when I get this issue again.

[corneliusludmann]

Cannot reproduce with the January release.

However, installing MinIO in an air-gap networks leads to a similar crash loop. That's why we configured MinIO to skip the client config. That should help here as well.

I'll close this issue now. Please reopen or ping me when this still occurs with the upcojming Feburar release.