proxy private ssh-only repo

[mrgleeco]

I understand the hack of switching to ssh from https via the .gitconfig insteadOf syntax.
However, in some cases, only SSH access is available to the repo. Is it possible to proxy a local private repo that does not have ANY http/s? It seems the request for the package <meta name="go-import"... is always necessary, as per golang module spec.

[linzhp]

We also have a similar need. We have a git server that only support SSH access. So the server doesn't support GET https://code.example.internal/foo/bar?go-get=1

We got similar issues with this stack question. With Git insteadOf config, we can do

GOPRIVATE=code.example.internal go get code.example.internal/communications/locales.git

But we cannot do

go get -v code.example.internal/communications/locales 
go get code.example.internal/communications/locales: unrecognized import path "code.example.internal/communications/locales" (https fetch: Get https://code.example.internal/communications/locales?go-get=1: dial tcp: lookup code.example.internal: no such host)

Is there a way for athens to translate code.example.internal/foo/bar into gitolite@code.example.internal:foo/bar and check out from git directly using SSH instead of calling go mod download or go get?

If not, @arschles @marwan-at-work, are you open to a contribution to add this feature?

[arschles]

@linzhp right now, Athens doesn't fetch code from VCS systems by any means other than go get. I believe that if you use an insteadOf directive in the Athens server-side git config (see https://docs.gomods.io/configuration/authentication/ for example), you can achieve this translation.

If you've tried this, and it doesn't work, we would absolutely open to a contribution to add this feature (I'd also like to see some more details on what went wrong 😄)

[linzhp]

The main issue here is go get, which always trying to get https://code.example.internal/communications/locales?go-get=1 first and expect a go-import meta tag in the response. The Git insteadOf translation comes after go get finds https://code.example.internal/communications/locales.git in the meta tag. Since our Git server doesn't support HTTPS at all, go get fails before hitting Git translation.

A solution I can see here is for Athens to do the HTTPS to SSH translation on its own and check out from Git using SSH directly, without using go get or go mod download. Do you think it viable?

[arschles]

@linzhp I see what you mean now, thanks.

You can configure Athens to respect GOPRIVATE on the server side. For example, you can do this:

$ export ATHENS_GO_BINARY_ENV_VARS='GOPRIVATE=code.example.internal'`

or set it in the config file:

GoBinaryEnvVars = ["GOPRIVATE=code.example.internal"]

And then Athens will set GOPRIVATE properly every time it does its go get

@linzhp @mrgleeco would that work for you?

[linzhp]

The issue is less about GOPRIVATE and more about go get. As long as Athens still runs go get, it will still use HTTPS protocol, which our VCS server doesn't support

[arschles]

@linzhp the gitconfig would be to prevent go get from using an HTTPS protocol, and the GOPRIVATE would prevent it from using any upstream module proxy. Would that setup not work if you do it locally? I want to see what Athens is missing

[linzhp]

When we run go get code.example.internal/communications/locales/pkg, it will request the following URLs:

GET https://code.example.internal/communications/locales/pkg?go-get=1
GET https://code.example.internal/communications/locales?go-get=1
GET https://code.example.internal/communications?go-get=1
GET https://code.example.internal?go-get=1

Most of these request will return 404 Not Found, but one of them, e.g., https://code.example.internal/communications/locales may return 200 with a meta tag https://code.example.internal/communications/locales.git. Then Go knows where the Git repo root is and call git to checkout that repo. Then if insteadOf is set, Git will use SSH instead of HTTPS.

However, if the VCS doesn't support HTTPS, the 4 GET requests will fail with something like

https fetch: Get https://code.example.internal/communications/locales?go-get=1: dial tcp 10.80.224.43:443: connect: connection refused

The GET requests were sent before calling Git regardless of insteadOf config. The exception is when we import a package as code.example.internal/communications/locales.git/pkg, Go knows where the repo root is, so it calls Git to checkout code.example.internal/communications/locales.git without sending the GET requests. From that point insteadOf is in effect.

As a workaround, we forked Athens to add a .git to the Go import path before Athens calls go get. Then after Athens gets the zip file, we rewrite the zip file to remove .git from the directory in the zip file

[arschles]

@linzhp I'm not understanding perfectly because you said above that when you run GOPRIVATE=code.example.internal go get code.example.internal/communications/locales.git, this works.

If Athens ran that same command on the server, would GOPROXY=athens.internal go get code.example.internal/communications/locales.git not work?

[linzhp]

To sum up: go get code.example.internal/communications/locales.git works, both on Athens and locally, go get code.example.internal/communications/locales doesn't work, neither on Athens or locally. Note there is a little .git difference in those two commands.

[marwan-at-work]

@linzhp @mrgleeco would a vanity import server like this one solve your use case? https://github.com/GoogleCloudPlatform/govanityurls

If not, I'd love to know why and I'm happy to consider potential additions to Athens for this.

[xytan0056]

@marwan-at-work thanks for the suggestion. I took a look.

1. Looks like we need to deploy it somewhere that supports accessible custom domain (eg. code.corp.internal). Our infra doesn't support it currently. Every service has to be service.corp.com. We still have the problem of having to rewrite import path.
2. I don't see govanityurls support regex. meaning we'll have to put every repo into vanity.yaml, which is not ideal for us to maintain.

[arschles]

@xytan0056 if the project that @marwan-at-work mentioned works for you, we will add documentation for when and how to use it. I'm looking forward to seeing how govanityurls works for you!

[xytan0056]

@arschles See my comment above. Unfortunately, it didn't work for our current infra. Moreover, we'll need to do this for every internal repo, which doesn't scale.

[arschles]

Sorry, I misunderstood your first comment. I'm not sure how we can make this work if you can't host a vanity server(s) at the same domains as your modules (e.g. code.example.internal). I'm still not sure if I'm completely understanding the issue here, though, so please correct me if I'm wrong 😄

[linzhp]

We had an internal discussion on this vanityurl idea. Here is what we found out:

1. "go get" uses HTTPS. It is almost impossible obtain an SSL cert for ".internal" domain within our infrastructure. So we can't deploy vanityurl on https://code.example.internal
2. We may be able to run a sidecar vanityurl within the same container as Athens, and override /etc/hosts to direct all "code.example.internal" traffic to 127.0.0.1 (localhost). However, we are still trying to figure out how to support HTTPS on localhost. Any ideas?
3. As @xytan0056 pointed out, we have a large number of repositories internally. Using govaintyurls means we need to edit vanity.yaml everytime people create a new git repo. We may have to fork govaintyurls to add the logic of mapping import paths to VCS repo roots

[arschles]

@linzhp regarding (2), using a self-signed cert with the sidecar is the only thing that comes to mind. it would mean that you'd need to add TLS support to govanityurls as well.

[rolandjitsu]

@linzhp I see what you mean now, thanks.

You can configure Athens to respect GOPRIVATE on the server side. For example, you can do this:

$ export ATHENS_GO_BINARY_ENV_VARS='GOPRIVATE=code.example.internal'`

or set it in the config file:

GoBinaryEnvVars = ["GOPRIVATE=code.example.internal"]

And then Athens will set GOPRIVATE properly every time it does its go get

@linzhp @mrgleeco would that work for you?

@arschles I'm having a similar issue and I've tried just this, but it seems like Athens is either not picking up on the env vars or it just doesn't work.

I've described my issue in #1618 but it might be similar to this one.

[R3n3r0]

Quando eseguiamo go get code.example.internal/communications/locales/pkg, richiederà i seguenti URL:

GET https://code.example.internal/communications/locales/pkg?go-get=1
GET https://code.example.internal/communications/locales?go-get=1
GET https://code.example.internal/communications?go-get=1
GET https://code.example.internal?go-get=1

La maggior parte di queste richieste restituirà 404 Not Found, ma una di esse, ad esempio, https://code.example.internal/communications/localespotrebbe restituire 200 con un meta tag https://code.example.internal/communications/locales.git. Quindi Go sa dove si trova la radice del repository Git e chiama git per eseguire il checkout di quel repository. Quindi, se insteadOfè impostato, Git utilizzerà SSH invece di HTTPS.

Tuttavia, se il VCS non supporta HTTPS, le 4 GETrichieste falliranno con qualcosa di simile

https fetch: Get https://code.example.internal/communications/locales?go-get=1: dial tcp 10.80.224.43:443: connect: connection refused

Le GETrichieste sono state inviate prima di chiamare Git indipendentemente dalla insteadOfconfigurazione. L'eccezione è quando importiamo un pacchetto come code.example.internal/communications/locales.git/pkg, Go sa dove si trova la radice del repository, quindi chiama Git per il checkout code.example.internal/communications/locales.gitsenza inviare le GETrichieste. Da quel punto insteadOfè in vigore.

Per ovviare al problema, abbiamo eseguito il fork di Athens per aggiungere a .gital percorso di importazione Go prima che Athens chiami go get. Quindi, dopo che Athens ottiene il file zip, riscriviamo il file zip per rimuoverlo .gitdalla directory nel file zip

hello but this fork is public ?

[linzhp]

No. After we migrate all Go code to a monolithic repository, this is no longer needed. So we are deleting the fork internally

[R3n3r0]

Ok thank you, but I don't think it's enough Modify the url for download surely the procedure to create the zipper must be modified too Right?