proposal: gno.mod resolve directive

[thehowl]

This issue is a proposal for adding a resolve directive to gno.mod, as a measure to achieve effective development across different gno.land deployments. It has arisen from considerations on #1306, which have spiralled into forming a proposal to solve this for any static analysis tool for gno development in general.

The resolve directory instructs the gno command line tool how to resolve dependencies, both on the local filesystem as well as remotely. It stems from the current problem we have of having two incompatible avl versions between GitHub and test3. This situation will arise again, so the key question is: how do I make sure my gno code is always referring to the code I intend to refer to?

Context

As it stands, the examples directory has packages and realms, whereby each contains its own gno.mod file. When gno.mod to all of examples was added, gno doc was changed to treat all of the directories within examples as "modDirs" (as long as they had a gno.mod): this is separate from stdlibs, which don't have a gno.mod at all, and have their import paths determined by their import path relative to $rootDir/gnovm/stdlibs.

On this last paragraph, note that if the local directory ., or any upper directory, contains a gno.mod, then that is also treated as a modDir, and loaded accordingly.

Now, the behaviour before #1306 for loading packages imported via gno.mod was the following:

1. Load each mod dir in order.
2. For all the dependencies of that mod dir, load the corresponding file in the cache.

This, in turn, was leading to the faulty behaviour experienced by Milos, Manfred, Guilhem and many others on the core team: gno doc avl was returned the cached AVL version, from any directory, rather than the up-to-date version in $rootDir/examples/gno.land/p/demo/avl.

This was caused by gno.land/p/demo/acl coming earlier in alphabetical order when reading the directory, which has a gno.mod containing gno.land/p/demo/avl. As such, gno doc went on to scan the cached directory, because it found it in a gno.mod file of one of its modDirs...

A first proposal

#1306 tried to fix the probelm by by changing the import loading for modDirs: the source code for each modDir is loaded first, with higher priority, and following that the imports from each gno.mod are processed.

The rationale here is that when a package imports another in examples, it actually probably refers to the latest version available in examples, not the one in the user's local cache; as such loading all the examples package should be done first before we try resolving any on-chain dependencies.

However, after some due consideration I realised this is not an appropriate approach to solve the problem, as it still considers the examples directories as modDirs, and as such will process their gno.mod files as "normal" dependencies. Say acl imports avl, and on-chain avl has a function Bad(), which has been removed in the updated GitHub version. But because it's on-chain (and cached on my laptop), if I do gno doc avl.Bad I will still get documentation for it, anywhere I am (even though it's not on the upstream anymore).

On the other hand, we also shouldn't resort to always loading dependencies from the examples directory as the default behaviour; if I'm writing a contract for test3, I want to have the ugly, old version of avl, otherwise trying to publish the code on test3 might not work. Note that defaulting to examples this would be the behaviour with #1306, as the examples' AVL is loaded before the dependency's cached code.

One approach might be to special-case examples, by not parsing its gno.mod imports; but this has problems because it still considers "examples" as special while we're trying to go into a direction of "any package and realm within could be on its own GitHub repo" (by having gno.mod for each file).

Another corollary is that if I have a root-dir and I am working on a project separate from it, my gno.mod importing gno.land/p/demo/avl will refer to the local, examples version, rather than the cached one downloaded on-chain. This behaviour may seem desirable, but if I'm making my package to target release on test4, I want to know what avl symbols are available in test4, not on the monorepo.

To get out of this mess,

I propose allowing users to specify how to resolve their dependencies, and have different caching strategies for the dependency. This would be specified through a gno-specific directive in the gno.mod file, which essentially specifies how to get their dependencies, with these three main manners in mind:

• A purely local experience, which wants to refer to packages in the examples directory, which is what the user is likely when they spin up a local gno.land node. It works by resolving gno.mod imports to $rootDir/examples/.
• gno.land, ie. the Portal Loop, which will be the place where most "live" development happens. More packages are available on top of the ones of the "local" experience, but occasionally, on backward-incompatible changes, some packages may disappear too, and some may be changed (if it comes as a result of a package being changed on the GitHub source).
  This would thus means the code is resolved to $GNO_HOME/pkg/mod@gno.land/ -- and it is explicitly checked if it is out of date when running commands such as gno mod download.
• test4.gno.land, and in general all other URLs, which are likely to have a snapshot of the examples packages (as well as stdlibs) at its launch, and their own set of published packages. Over the course of a couple months, the packages may become obsolete on mainline development, have old features or bugs, or lacking some.
  This would thus means the code is resolved to $GNO_HOME/pkg/mod@test4.gno.land/, and can be cached forever (as the packages and realms, at least for now, cannot be updated.)

In order to create further confusion and continue on the well-established tradition started by the Go project¹, I propose to name this directive resolve, though I'm happy to make this target as well if met with opposition ;)

resolve gno.land
resolve test4.gno.land

// can specify any hostname, even localhost:
resolve test999.howl.moe localhost

// this one resolves to `$rootDir/examples`:
resolve local

// prefer local, fallback on gno.land:
resolve local gno.land

The default behaviour, when a resolve directive is not specified, would be the last: try to resolve the packages locally (works only if a root-dir is specified), and if they're not available try fetching them from gno.land ("staging" portal loop testnet).

As a consequence:

• examples/ packages and realms would always be specified as resolve local, as they want to refer to their root-dir cutting edge versions.
• Most local developers will want to use resolve local gno.land, so that the resolved dependencies point to what will be on their local testnets; but they can also import packages from the gno.land portal loop testnet.

cc/ @harry-hov

Footnotes

1. see gomod ref, which has already directives require, replace and retract, all starting with re- :P ↩

[harry-hov]

I propose allowing users to specify how to resolve their dependencies,

I agree with you 💯

--

But. I imagine gno.mod file to majorly declare pkg name, dependencies and versions (dependency management). I'm bit inclined towards not having resolve directive in gno.mod file. As at some point we might want gno.mod to make sense onchain too.

What are your thoughts on providing some configuration options or flags in the tool itself?

Just like we have -remote in mod download command:

gno mod download --remote=<remote>

--

gno doc avl --remote/resolve="test3.gno.land"
gno doc avl.Bad --remote="local"  // default: gno.land?

??

--

Maybe half of the problem will be solved when we will support pkg versioning :)

[thehowl]

Will make another, different proposal