[proxy] Replace nginx with Caddy

[aledbf]

Changes:

• Migrate proxy from NGINX to Caddy
• Remove bash script to build configuration files
• Remove chart configuration files
• Cleanup chart variables
• Remove the option to use HTTP or HTTPS and make HTTPS mandatory.
• Migrate lua scripts to Caddy middleware (Go)

TODO:
- make minio and docker-registry optional
- Private workspace ports

• Custom 404 error pages?

fixes #3611

[aledbf]

/werft run

👍 started the job as gitpod-build-aledbf-proxy2caddy.27

[aledbf]

/werft run

👍 started the job as gitpod-build-aledbf-proxy2caddy.55

[aledbf]

It works very nice now! What was the issue?

There is an issue in Go related to http/2 golang/go#42534 (comment)
(temporal workaround is to disable http/2). TODO: check if this affects ws-proxy when is exposed directly (pluggable workspaces)

[akosyakov]

There is an issue in Go related to http/2 golang/go#42534 (comment)
(temporal workaround is to disable http/2). TODO: check if this affects ws-proxy when is exposed directly (pluggable workspaces)

I see, wonder whether it can affect end users? e.g. if one tests an app loading time on a port location then he can get wrong impression because of disabled http2, i.e. resources will be loaded one after another, not parallel? or it does not affect them?

[aledbf]

I see, wonder whether it can affect end users?

Maybe? I can perceive a difference only opening something like vscode (~2000 request every time)

resources will be loaded one after another, not parallel?

There is a default of 6 parallel downloads. This is really old but the values are similar https://www.stevesouders.com/blog/2008/03/20/roundup-on-parallel-connections/

[akosyakov]

Unfortunately it makes really huge difference from perf perspective. Our website loaded with slow 3g with disabled caching in production vs the PR:

The thing is that optimisation techniques with http2 and without are quite different, and not having http2 on port location will harm usefulness of Gitpod in this sense.

[aledbf]

/werft run

👍 started the job as gitpod-build-aledbf-proxy2caddy.142

[geropl]

The thing is that optimisation techniques with http2 and without are quite different, and not having http2 on port location will harm usefulness of Gitpod in this sense.

Hm. I'd really love to see us move away from nginx. But it'd would also nice to not have to compromise. @aledbf Would it make sense to wait just a little longer and check if either:
a) golang/go#42534 gets fixed upstream
b) caddy uses the fixed net/http code (this looks like it already does...?) But the question is if we really want to go with untested code here.

Personally I'm inclined to wait: As nice as a switch would be it's not blocking us AFAIK, and the nginx mess stuff is stable atm. Also, the PR is not likely to rot fast. WDYT?

[aledbf]

a) golang/go#42534 gets fixed upstream

Agree.

b) caddy uses the fixed net/http code

I tried the proposed fix without success.

Personally I'm inclined to wait:

Yes, I agree with that. That is why this PR is still a draft.

Edit: after all these comments, I updated the PR with a fix that seems to work 🙃

[aledbf]

@akosyakov please try again

[aledbf]

[akosyakov]

works fast and stable for my use cases, but i did not check code

[akosyakov]

@aledbf one thing which seems to be wrong it is how long does it take to download global.css with new PR in our website on slow 3g connection without caching again the prod. It is still slow compare to prod with recent changes.

[csweichel]

Can we build this caddy image as part of the Gitpod repo, or is this a fork of caddy?

[aledbf]

This image is a fork of Caddy with the updated packages aledbf/caddy@68fbb8e
I can do the same thing in the gitpod org, forking caddy and adding the commit, or maintain the code in my fork until the fix land in Caddy.

[aledbf]

/werft run

👍 started the job as gitpod-build-aledbf-proxy2caddy.155