[self-hosted] Download of workspace failed with self-hosted / minio (at least with gitpod-k3s)

[corneliusludmann]

Downloading of workspace content from the workspace history list does not work. You simply get a /sorry page with “Error”. That's what the proxy logs:

2020/12/17 08:31:24 [error] 125#125: *1512 lua entry thread aborted: runtime error: access_by_lua(lib.locations.conf:107):3: http2 requests not supported yet
stack traceback:
coroutine 0:
	[C]: in function 'capture'
	access_by_lua(lib.locations.conf:107):3: in function <access_by_lua(lib.locations.conf:107):1>, client: 10.42.0.1, server: gitpod.dev.ludmann.org, request: "GET /workspace-download/get/a45cdd20-931d-4604-86bb-ac7e56071596 HTTP/2.0", host: "gitpod.dev.ludmann.org", referrer: "https://gitpod.dev.ludmann.org/workspaces/"

After removing http2 from vhost.server.conf this error disappears. However, workspace downloads still not work. Instead, you get a /sorry page that says 404. The var ngx.var.targetUrl in lib.locations.conf looks good and points to the minio pod. At least form the proxy pod, the targetUrl is reachable via curl (but gives 403 - unauthorized).

[corneliusludmann]

I finally found the actual problem (besides the http2 thing):

The proxy tries to connect to:
http://minio.default.svc.cluster.local:9000/gitpod-prod-user-025937da-b2e5-487e-b59e-c5d3eec5d179//workspaces/eb79b0fd-22c2-41f5-93fb-8605e2b3f831/full.tar

But the user bucket is gitpod-user-025937da-b2e5-487e-b59e-c5d3eec5d179 (without prod).

The problem is here:

gitpod/components/server/src/workspace/workspace-download-service.ts

Lines 52 to 64 in f3a89f1

	// we must harmonize this with https://github.com/TypeFox/gitpod/blob/8fc0c82a55da1ca4b5f6ab61deb9c9cd49eff644/components/ws-daemon/pkg/storage/storage.go
	// Beware: do NOT use env.kube_stage which has some "legacy" translation mechanism which doesn't fit the ws-daemon mapping.
	const stage: string = ({
	"production": "prod",
	"staging": "prodcopy",
	} as any)[process.env.KUBE_STAGE || ""] || "dev";
	const bucketName = `gitpod-${stage}-user-${wsi.ownerId}`;
	const path = `/workspaces/${workspaceId}/full.tar`;
	const signedUrl = await this.storageClient.createSignedUrl(bucketName, path, "read", {
	promptSaveAs: `${workspaceId}.tar`
	});

But also here, I guess:

gitpod/components/server/src/storage/commons.ts

Lines 14 to 17 in f3a89f1

	export function getBucketName(userId: string, stage: KubeStage): string {
	const bucketPrefix = getBucketNamePrefix(stage);
	return `gitpod-${bucketPrefix}-user-${userId}`;
	}

Or even here:

gitpod/components/content-service/pkg/storage/minio.go

Lines 239 to 241 in f3a89f1

	func minioBucketName(ownerID string) string {
	return fmt.Sprintf("gitpod-user-%s", ownerID)
	}

gitpod/components/content-service/pkg/storage/gcloud.go

Lines 651 to 653 in f3a89f1

	func gcpBucketName(stage Stage, ownerID string) string {
	return fmt.Sprintf("gitpod-%s-user-%s", stage, ownerID)
	}