[RFC] Git integration for decentralized OSS and mixed go-imports (bonus)

[moul]

Disclaimer: This is an early issue and not a current priority. However, with new team members and previous discussions, I wanted to create an official forum for future discussion.

Context: recently, we welcomed @ajnavarro to our team. Before joining us, he worked on the development of go-git. Another member of our core team, @harry-hov, has contributed to @git in the past and is now focused on porting go mod to gno mod by fetching gno dependencies from the chain, which differs from the way go uses git for this purpose.

Three initial exploration options:

• To enable the go import of a gno contract with automatic transpiling of .gno files.
• Same, but with generated type-safe Golang clients.
• Degen: By using Golang's vanity URL feature, we'll switch to tendermint.org/gno to boost censorship resistance and migration flexibility. Exploring a git interface to deliver go sources via gnoweb, gnoland, or a standalone service is possible, but probably unnecessary.

[ajnavarro]

I had been looking into this, and maybe we can follow a different approach:

go mod and go get are using a special proxy called proxy.golang.org, here is the spec: https://go.dev/ref/mod#goproxy-protocol

We can implement a special proxy that detects a specific import URL, like gno.land , github.com/gnolang/gno, or tendermint.org/gno, and redirect the source code fetching and transpilation to be done using the Gnolang network. If that is not the case, proxy.golang.org is called. This can be done by deploying our own proxy server at proxy.gno.land , or starting it up locally to avoid censorship. The user has to change the GOPROXY env var to point to this server.

We can also serve transpiled go code depending on the import path; if github.com/gnolang/gno is called, we can use the Gnoland network to find the specified package and transpile it. If gno.land is used, we can return .gno files directly.

Edit: I realized that we can ignore the step of querying the default proxy.golang.org server, because GOPROXY is a list. so we can modify the env variable to something like: GOPROXY="https://proxy.gno.land,https://proxy.golang.org,direct" or GOPROXY="https://localhost:8888/proxy,https://proxy.golang.org,direct"

@moul WDYT?

[ajnavarro]

gnomodproxy PoC here: gnolang/gno#727

[moul]

Great! I like the idea of using the official proxy feature to make it transparent from a Go perspective.

We could reuse a similar approach for gno mod (cc @harry-hov), possibly linking it with infrastructure tools such as custom indexers built by @ilgooz's team.

[thehowl]

Hmm. It's not clear to me from this conversation what the significant advantage to proxying would be.

As far as I know, proxy.golang.org exists and makes sense because large Go repositories tend to build over time complex dependency graphs and the time for a first go get increase linearly -- the proxy largely solves this issue.

I think that at the scale that we are now, and because there are currently no realms pulling other packages and sources from hundreds of different sources, a proxy would be a solution to a problem we don't have, adds a (default) chokepoint/SPOF (which can be disabled, but I imagine would still be active by default), and poses a problem of trust (ie. that the contents in the proxy are untampered, up-to-date and match the original source's).

I'm probably missing something and want to make clear that I'm very much open to a change of heart; but I'd like to better understand what the reasoning behind adding a proxy, and if so at this stage, would be.

As for the automatic transpiling; I think this should not be done at the proxy-level, but rather it should be done at the gnoweb/website level, so that working with Gno code from Go works out-of-the-box, without setting up a proxy. For instance, when I make a request to https://gno.land/p/demo/avl?go-get=1, I get the import meta tag:

<meta name="go-import" content="gno.land/p/avl git https://test3.gno.land/gitproxy/p/avl">

So we would have a way to download the "repository" at https://gno.land/gitproxy/p/avl, implementing in either the smart or dumb protocol the git http protocol. I would imagine this repository as always having 1 commit, representing the internal status on gno.land, done with a fixed message and timestamp (given two equal package/realm contents) so that the repository doesn't change if the content don't change.

We could also make this transpile to Go code for packages, and to Go code but using SDK clients for realms.

[ajnavarro]

@thehowl The idea was to use the gomodproxy protocol, not to create a specific proxy itself. It is basically the same as using the HTTP git protocol. I think we will have some advantages doing that approach instead of downloading directly from git endpoints:

• We can authenticate Gno source code in the future using the checksum database specs.
• We can implement a package index that will mimic Go existing tools by default: https://index.golang.org/index
• The proxy protocol is much simpler to implement than the git HTTP protocol. No need for packfiles, hashing, refs...

So, the idea is instead of using a git repo URL from your example: https://gno.land/gitproxy/p/avl (I think the URL should be https://gno.land/gitproxy/gno.land/p/avl) we can just do http://gno.land/gnomodproxy/gno.land/p/avl/@v/v0.0.0.zip

[ajnavarro]

I would imagine this repository as always having 1 commit

I think that cannot be the case, and we should have at least version tags to support gno.mod in the near future, something that is already solved if we use the gomodproxy specs.