Proposal: package management #851
Comments
|
Original reply by @verdverm in cuelang/cue#851 (comment) Generally like that Go's module system is the inspiration and that much can be copied over. However I have some concerns about reusing the
This seems like it will prevent many use cases for CUE + Go dependencies in the same project. If I have a repository which is both CUE and Go, and the set difference of their respective dependencies is non-empty, then wouldn't the respective Am I missing something here? |
|
Original reply by @myitcv in cuelang/cue#851 (comment)
The reasons are exclusively attributable to re-using the proxy.golang.org and sum.golang.org infrastructure. The answer to "Why have you chosen to use {proxy,sum}.golang.org?" has the rationale.
At least as I envisage it, Perhaps I should be more explicit about this above? I had hoped to stash this file out of the way in |
|
Original reply by @nyarly in cuelang/cue#851 (comment) I can't say that I receive the idea of reusing the Go modules scheme with delight. In principle, MVS centralizes the control over the version of a package I'll use in my project; I've had a lot of success in lockfile-driven systems being able to exclude releases that were incompatible with my usage. MVS also assumes that package authors want to and and able to perfectly conform to semver; if the best intentioned authors sometimes break interfaces with a release. Finally, MVS makes strong assumptions about the quality of releases, but then defaults to a source-repo-as-package-repo approach. In practice, I haven't found that vgo has enabled large Go projects the way that it was intended to. More and more, I work with container orchestration, and both the Kubernetes and Docker projects are a challenge to incorporate via gomodules. Incorporating version resolution into every Go task leads to a lot of confusing and frustrating issues. My intuition is that only local changes should influence the outcome of a build, but post gomodules, I find that any of my dependencies can break things. Personally, I was having great success with the "official experiment" of godeps. It was a familiar and well-worn approach with version satisfaction and lockfiles. The process by which it was overturned felt (as a daily user of Go) authoritarian and ill-considered. It's possible that CUE avoids a number of these issues because it doesn't have a large legacy base to support, but I wanted to give voice to my trepidation. |
|
Original reply by @verdverm in cuelang/cue#851 (comment) In order for CUE to maintain both, will it be reading the Go code and resolving used imports? Is that even something CUE should be doing? Are there examples of other languages where they are crossing dependency management or repurposing dependency files? Do we need a module proxy for a first module implementation? I'd argue that Go's ecosystem had been quite vibrant long before it had this. Having the sumdb for security & reproducibility is a lot easier. ( w.r.t. the current sumdb format, it's a line oriented file & parser and super simple, though having it specified in CUE would not impede reusing the existing code |
|
Original reply by @verdverm in cuelang/cue#851 (comment) @nyarly I would say that MVS is a selection algorithm with deterministic results and that Go added some assumptions that align with the Go 1 compatibility guarantees. Go's module system does have exclude and retract clauses (https://golang.org/ref/mod#go-mod-file-exclude). I think the semver topic is orthogonal to the MVS vs Lockfile, as npm / yarn dependencies are often specified in semver format. There are authors in all languages that will break semver rules. I don't think the dependency selection method helps here. Anecdotally, as a daily Go user too, I welcomed the addition of the |
|
Original reply by @verdverm in cuelang/cue#851 (comment) @myitcv in the section "Changes to cmd/cue" there are several references to a |
|
Original reply by @myitcv in cuelang/cue#851 (comment)
Thanks for that catch. The proposal is that CUE's vendor directory will be |
|
Original reply by @nyarly in cuelang/cue#851 (comment) @verdverm Tempted though I am, I don't mean to re-litigate the go modules design here. With regards to CUE though, my impression is that adopting much of that design is settled. If that's so, I don't want to foster acrimony by questioning those decisions here. |
|
Original reply by @morlay in cuelang/cue#851 (comment) New If we just want to get the at least, we may need github will return gitlab only handle |
|
Original reply by @morlay in cuelang/cue#851 (comment) Another question Could we add non-cue requirement in
|
|
Original reply by @verdverm in cuelang/cue#851 (comment) I believe Auto-detect -> gen would break when I have repos which are both CUE and Go modules (I have several already and know that the "Go" files found would not parse (they are actually It does seem nice to have a way to |
|
Original reply by @myitcv in cuelang/cue#851 (comment) @verdverm. Thanks for your comments/feedback.
Yes, in the case of a Go + CUE module,
This is of course the fundamental question when it comes to this proposal. The arguments in favour and against are laid out in "Why have you chosen to use {proxy,sum}.golang.org?". If there are points you feel are missing, or need stressing more clearly please let me know.
This is not disputed. However, there are real benefits to using a proxy and sumdb. As I mention about though, it's whether those benefits outweigh the costs.
I'm not clear what point/position you are advancing here. sum.golang.org fulfils a specific role, namely to fill in the gaps if a module does not already have a
Again I'm not clear whether you're arguing in favour of the current proposal to use a |
|
Original reply by @myitcv in cuelang/cue#851 (comment) @morlay - thanks for your comments and feedback.
I think you might have missed the bit in the proposal that talks about exactly this fallback:
So any existing code host that replies to The loader would also, like Hopefully that allays your concerns on this point?
I'm not sure this would help, because other package management systems would then have to learn to read Instead, per cuelang/cue#646, The alternative to generating from Go/other source code (and the tooling dependencies that implies) is for those projects to pre-generate CUE, and for the dependency to be on the CUE module. In which case no dependency on tooling etc exists.
Apologies, I'm not entirely sure I follow what you are saying about extractors. I think (and please correct me if this is wrong) you are suggesting that steps could automatically be run when adding/loading a package dependency. If so, I don't think that's a direction we want to head in. The |
|
Original reply by @myitcv in cuelang/cue#851 (comment)
But in such a scenario of there being a specific CUE module (thereby obviating the need to generate from Go source) then there would be no requirement of the user of that module to have a |
|
Original reply by @verdverm in cuelang/cue#851 (comment) When it comes down to it, I do not want CUE touching my Go, (
I fail to see how this statement aligns with the proposals idea for CUE modifying Go module files. Is this not what the In early conversations about CUE modules (iirc) @myitcv said that a first implementation would not have all of the bells and whistles. I think that any use of |
|
Original reply by @verdverm in cuelang/cue#851 (comment) Some more questions on possible conflicts with CUE's use of
Given a dual module (both Go and CUE), I see some issues with supporting the following:
|
|
Original reply by @verdverm in cuelang/cue#851 (comment) Another possible edge case:
Essentially there is a dependency which needs a |
|
Original reply by @verdverm in cuelang/cue#851 (comment) Scenario:
But I don't see how the I suppose any CUE official k8s could have a different module path, but not necessarily required given replaces are possible I think there are more situations around replace than those I outlined above, outside of a "dual module", where both a CUE and Go module share the same path, and the Go module is only using CUE in CI, not for part of the code |
|
Original reply by @morlay in cuelang/cue#851 (comment) @myitcv Sorry. I missed
I understand this concern. In my hack, i use new attr But this is not work for other non-vcs-based pkg managment, like npm. I agree, we prefer to use
I implement another extractor https://github.com/octohelm/cuemod/tree/main/pkg/extractor/golang to generated cue defs from go codes without using so Howerver, we need
Yes. it is. the example flow in my hack tool: Howerver, I think generating & pushing somewhere, and importing generated code is better. @myitcv Thanks again. |
|
Original reply by @mpvl in cuelang/cue#851 (comment)
A potential embed functionality should not be allowed to embed cue.mod or any subdirectory that crosses a module boundary (like in Go. If I'm not mistaken, this edge case cannot occur with these restrictions. |
|
Original reply by @mpvl in cuelang/cue#851 (comment) @verdverm If it is desirable for the CUE and Go to be versioned independently for external, it is generally a good idea to put it in separate repos. Note that tagging would otherwise get a bit messy, if nothing else. Also not that for importing only, that is if the CUE in a module is not to be shared, but if one still wants to include other modules, it is possible to use an "anonymous" module. Anonymous modules do not require a top-level |
|
Original reply by @mpvl in cuelang/cue#851 (comment) @nyarly I'm not wedded to semver at all, and neither was MVS IIRC: if there are no backwards incompatible changes, having one version is really enough (see also https://www.youtube.com/watch?v=oyLBGkS5ICk). The reasons semver was chosen is because it was a de facto standard in the community and there needed to be something. With the special interpretation of major version |
|
Original reply by @verdverm in cuelang/cue#851 (comment) @mpvl My understanding is that Personally I don't see this as "co-sharing", because Go has no say in this, it is a unidirectional CUE choice. I am unaware of any other situations like this. It is unusual and feels like an anti-pattern or bad hack. This is the opposite of my experience with other CUE decisions, which have taken a thoughtful and paced approach. Go took 10 years to include PROXY / SUMDB. Why are these services required for a first implementation? Generally, what I don't like as a language A user is some other language B messing with my lang A dependency files to merge in its own. Especially when it has its own already. Then to add an extra file to my project because it wants to do this, so that lang A toolchain doesn't undo lang B's abuse, this is not desirable. There are conditions where this will break and it forces lang B's choices for code organization. The proposal does not state where this extra file will live, but given that people have all sorts of ways they organize code, it's likely to make someone unhappy. Question wearing a security hat: Could an unneeded lang A dep (think malicious package) be more secretly included via a I find the proposed benefits / drawbacks incomplete and several complications have not been addressed. The claim that doing this Missing tradeoff points:
|
|
Original reply by @mpvl in cuelang/cue#851 (comment)
Of course if this is true, then the same issue exists with using any combination of repos. The thing is that for most package managers, reproducibility isn't really a feature anyway, so having a few more version skews seems like the lesser problem, or at least something that justifiably should be tracked manually. With MVS, we have the possibility of addressing this automatically. It seems to argue even that Go's MVS system should be pluggable.
We are looking at the problems at hand, rather than what is conventional. I can't say I'm thrilled with approaches taken by most other languages (there are some notable exceptions, like Nix). MVS seems to fit the constraints of configuration and CUE particularly quite well and piggybacking on Go not only is convenient, but also avoids some bad dependency skews that are otherwise impossible to solve. Given this reality, though, I do think the design needs to change a bit.
Those 10 years were not without battles and desperation of not having dependency management. There is no need to repeat that history if we have something good now.
There is no need for that. It can be accomplished by generating a Go file capturing cue dependencies hidden in the
The
I don't really don't see how this setup can influence the outcome of existing Go code other than forcing a newer version. |
|
Original reply by @verdverm in cuelang/cue#851 (comment) I have a
Remote PROXY / SUMDB are not required for dependency management, they are optimizations for package cache and a "trusted" hash store. My understanding is that reusing these are the sole reason for using a
Are you saying CUE is now intending to use I have been using MVS dependency management with both CUE and Go in the same repos, successfully, without commingling. (hof has a reference implementation that uses MVS without commingling, that includes sumdb calculations, without the SUMDB remote, no PROXY needed) There are many organizations with polyglot repositories that get along just fine without mixing language dependencies and systems.
Besides increasing download and build times, forcing newer versions is an important point and a source of malicious dependencies. This is a security concern. By creating a dependency system that starts in Go, can then jump to processing CUE only dependencies (via a
I am also a fan of MVS and reusing this scheme in CUE. My disagreeability is specific to CUE using
This is in fact what |
|
Original reply by @verdverm in cuelang/cue#851 (comment) RE: embeds Go allows embeds within a module, and then a Go binary that imports that module will have them as well. Trying to work through an example, my Go program has some dependency D. In the case that...
Now if the maintainer of D has not vendored their CUE code (including Does that sound correct? |
|
Original reply by @verdverm in cuelang/cue#851 (comment) RE: language jumping via intermixed dependencies Maybe a more succinct way to describe my concern here is that two independent Go sources (Module A does not have any dependency path through Go code to Module B) can nonetheless have a version bumped through a CUE module to B, due to a common Go dependency and MVS. I believe only one dual module is required to link the dependency DAGs and force A to include B in its MVS calc. That dual module does not need to use CUE and Go together (CUE for CI only). Given enough widely used modules in each language start connecting, this could have a significant impact in the number of dependencies accumulated through an intermixed MVS and the subsequent versions. |
|
Original reply by @mpvl in cuelang/cue#851 (comment) We're working on a new proposal that separates concerns a bit more and explains some of our thinking.
Overall I don't share that concern. In fact the opposite. If the bump occurs because Go and CUE are co-versioned, then there is, or at least may be, a dependency. But we're going in circles here.
Not saying that is not possible, but there are certainly cases where not commingling will result in badness. Do you have a documentation of how MVS in hof works? |
|
Adding a drive-by comment, purely so I don't forget. But in the context of being able to import/embed JSON Schema (#438 and the as-yet unraised embedding proposal) I think we will need to consider CUE module dependencies that do not contain a Clearly this concept goes beyond just JSON Schema to other encodings, but in the context of cue-lang/vscode-cue#12 this was the example foremost in my mind. |
|
Adding a drive-by comment, since the thoughts are in my head... It would be great if the module system was something that other tools building on the Go API could import and use.
|
|
@verdverm - we definitely intend for However it's very likely the case that, like |
|
Drive by comment on dotless module paths to ensure we revisit this later: https://go-review.googlesource.com/c/website/+/359594 |
|
Drive by comment for
Most of this I expect would come over depending on the parts of the Go implementation that are copied in and when that happens.
I agree, adding a module should be intentional via a |
|
@verdverm yes, the CUE module design will benefit from all the features of the Go module design. Workspace support is something I am keeping an eye on. It's been through a number of rounds of discussion and design as part of the Go tools working group. Needless to say, the priority will be getting full package management/module support first. |
|
Noting per #1494 that something like |
|
cue/run would be another great candidate to consider for |
|
to be clear, I'd like to be able to do something like |
|
Modules will have a local cache, much like Golang does, so it should work in any directory. I would expect that removing the need to be in an existing module is a matter of code paths and arg parsing. This seems to be inline with |
|
@myitcv What’s the current status on this proposal? At Dagger we want to prioritize package management since it’s one of the most requested features. How can we help? |
|
👋 Just doing another checkup. I'd really love to expand the usage of Cue within the company I work for, however having to brew up a manual install script to get our shared packages setup just right is not going to be viable in the long term. A builtin solution would make Cue so much more frictionless to use :) |
|
We have created a revised package management and modules proposal that is linked to and discussed at #2330. |
myitcv
added
modules
Originally opened by @myitcv in cuelang/cue#851
With extensive inputs from @mpvl.
Proposal summary
We propose adding package management to CUE, using an approach analogous to Go using Minimum Version Selection and semantic versioning. The changes are limited to
cmd/cueand thecue/loadpackage, and hence have no bearing on CUE the language.As an interim measure, we propose using proxy.golang.org as a module mirror, and the checksum database sum.golang.org for authentication, until such time as the CUE project can host such services itself. Use of these services will be enabled by default for
cmd/cueandcue/load, but entirely configurable, just likecmd/go.The proposed approach is broadly identical to the approach followed by Go modules. Indeed this proposal borrows heavily from Russ Cox's blog posts introducing
vgo, the core Go modules documentation, and interactions with Bryan Mills and Jay Conrod from the Go team.For those who might be less familiar with Go, in particular Go modules, for the more significant parts of this proposal the relevant parts of the Go module reference have been copied and adapted for the CUE context to save jumping between multiple documents. For less significant parts of the proposal, or those for where the concept is truly identical between both approaches, then we generally choose to link to the relevant Go documentation.
I understand Go modules; what's the TL;DR?
Here is an abridged version for those who have a working knowledge of Go modules, presented from the user perspective with various implementation considerations thrown in where relevant.
What will be different?
cue get gotocue import go- see cmd/cue: rename 'get go' to 'import go' #646.cue.moddirectory; the root of a Go module is identified by ago.modfile.go.modfile that mirrors thecue.mod/module.cuefile.cue.moddirectory without amodule.cuefile); in Go this is not possible.cue.mod/module.cue; this file is semantically equivalent togo.mod, with all directives supported albeit via a different syntax.cue.mod/sum.cueas its equivalent ofgo.sum, but in a CUE format (for consistency withcue.mod/module.cue).cue.mod/pkg, as it is today.cue/loadpackage for loading CUE instances. Go has no equivalent, and instead relies ongolang.org/x/tools/go/packagesto wrapcmd/go. Working with CUE modules therefore does not require the user to havecmd/cueinstalled.cue/load.Configconfiguration type is extended with the necessary fields for controlling module resolution and loading behaviour; these are essentially the backends to the frontend flags and environment variables available incmd/cue.cue-import<meta>HTML tags with a fallback togo-import.What is the same?
$HOME/cue/modcache, and be configurable viaCUEMODCACHE.cmd/cuewill have an analogous set of module-related commands ascmd/go::cue getcue list(andcue list -m)cue mod downloadcue mod editcue mod graphcue mod initcue mod tidycue mod vendorcue mod whycue envcmd/go's module-aware command flags and environment variables:GO*environment variables have equivalents of the formCUE*,-modhas a CUE equivalent--mod,-modfilehas a CUE equivalent--modpath.--mod=vendorremains an option. The existence of a vendor directory (cue.mod/pkg) implies--mod=vendor. This approach represents a backwards compatibility mode for how things work today.@latestet al).Background
The module and package concepts of CUE are directly inspired by their equivalent in Go. It is natural therefore to consider package versioning following the same model as Go.
One important and pleasant fact to note is that because CUE has already established the concept of a module, it does not have to manage a large legacy pre-modules world, unlike
GOPATHin the Go project. This is reflected in some of the decisions in this proposal. As a result, we do not need to distinguish between module-aware mode non-module-aware mode: everycmd/cuecommand is and will remain module-aware by definition, the same is true forcue/load.The issues
As noted above, CUE has already established the concept of a module. The root of a module is denoted by a directory that itself contains a
cue.moddirectory. The contents of this directory are mostly managed by thecuetool. In that sense,cue.modis analogous to the.gitdirectory marking the root directory of a repo, but where its contents are mostly managed by thegittool.Here is a minimal example that declares a CUE module
example.com/blah, a packageblahwithin the root of that module, such thatblahimports a third-party CUE packageacme.com/quotethat is vendored withincue.mod/pkg(written in thetxtarformat):From the root of the module we can
cue eval:Because there are no arguments passed to
cue eval, the implied argument is., the package in the current directory.cmd/cuethen hands off tocue/loadto resolve and load..In this example,
example.com/blahis referred to as the main module. As we can see from the packageexample.com/blah, it has a dependency onacme.com/quote, a package that is not part of the main module.cue/loadcurrently uses the simple rule therefore of searching for all dependencies outside of the main module withincue.mod/{pkg,gen,usr}directories, unifying the result.In this instance, the
acme.com/quotepackage is not part of a module, but it could well be.So whilst
cmd/cueautomatically takes care of resolving imports or package paths for us (viacue/load), everything else is left to the CUE developer. Vendoring of packages withincue.mod/pkghas to be done by hand, there is no mechanism by which dependencies can be fetched from source code hosting sites or remote repositories.For an initial implementation of
cmd/cuethis was more than sufficient. Indeed, users have adapted shell scripts to help with creating minimal vendors, and thehoftool has support for building such a vendor viahof mod vendor.But such an approach will neither scale for a larger CUE user base, nor an ecosystem of tools built on top of/with CUE and the
cuelang.org/go/...APIs. Specifically such an approach:Reimagining our example above, we want to:
acme.com/quote, without needing to worry about how and where we fetch that code;example.com/blahis reproducible;cmd/cueto control the dependencies of the main module, including automatically fetching CUE modules when required;Requirements
The following requirements have driven our thoughts on why and how to add package versioning to CUE.
Package versioning in CUE must:
cmd/cueorcue/load: simplicity, speed, and understandability. In general, version management work must fade to the background, not be a day-to-day concern;For the remainder of this document we borrow slightly adjusted definitions of the following terms from the
vgoproposal:One important difference from the Go modules implementation is that the CUE implementation should not live entirely in
cmd/cue(as it does incmd/go). Instead the bulk of the implementation will lie withincue/load, with a change tocmd/cueeffectively being the means by which the main module and its dependencies are controlled from the command line. That way, existing users of thecuelang.org/go/...APIs can continue to load and work with CUE instances, enjoying the same benefits that a seamless module experience will bring tocmd/cue. For tool authors who use the API but also requirecmd/cue-esque control over CUE modules and dependencies, thencuelang.org/go/cmd/cue/cmdcommand instances can be created and run without requiring the user of their tool to have also installed CUE, as is the case today.A key building block in CUE is the Go compatibility promise:
Correspondingly, this proposal adopts exactly the same concept of semantic import versioning introduced with Go modules, and with it the import compatibility rule for CUE:
Detailed Proposal
This section gives a brief overview of the proposal. Details are presented in the next section.
Throughout the rest of the proposal, the term "the loader" generally refers to the loading of modules and packages that happens through the use of
cmd/cueorcue/load.Rename
cue get gotocue import goIn preparation for full module support in CUE, we will need to repurpose an existing
cmd/cuecommand, renamingcue get goit tocue import go. The detail of this is covered in cuelang/cue#646.This change fits nicely with a recent change to the current semantics of
cue get go. Prior to a616925,cue get goattempted to resolve its package arguments that were not fully satisfied bygo.{mod,sum}automatically, via the use ofgo/packages(which itself usescmd/go). This would result in changes togo.{mod,sum}.With a616925 we have instead shifted to a model of requiring that a Go dependency of
cue get gocan be fully resolved viago.{mod,sum}without requiring further changes to either. This aligns with the new Go 1.16 default ofcmd/gobuild commands assuming a read-only default.This also aligns well with the new name
cue import go: unlikecue get go, the command does not imply any fetching or resolution.cue import gowill therefore fail in case any of its arguments cannot be fully resolved via the currentgo.{mod,sum}, and the user will need to rungo get -d(or equivalent) to ensure full resolution is possible.As part of this rename,
cue import gowill be changed to generate into thecue.mod/imphierarchy:cue get goincorrectly generates files within thecue.mod/pkghierarchy.Declaring module dependencies
At the core of this proposal is the ability for a CUE module to declare dependencies on other CUE modules through versions. This will be done by an extended schema of
cue.mod/module.cuefiles. Expanding our our example from earlier:Note:
example.com/blah/v2. Like Go modules, CUE modules adopt the concept of semantic import versioning. In this case, our move to a new major version indicates that we have made breaking changes compared toexample.com/blah, and this is reflected in the module path. A consumer of theblahpackage at the root of this module would therefore writeimport "example.com/blah/v2".acme.com/quotefrom thecue.mod/pkgdirectory, and instead replaced it with arequirefield in our module definition. Cryptographic sums incue.mod/sum.cueallow us to verify the contents of theacme.com/quotemodule whenever it needs to be downloaded.go.modfile that looks very similar to thecue.mod/module.cuefile now exists at the root of the CUE module. At least initially it is planned to host CUE modules on top of the existing Go modules infrastructure, specifically the module mirror at proxy.golang.org, and the checksum database at sum.golang.org. Declaring such a file is necessary for us to satisfy the GOPROXY protocol and existing checksumming mechanisms. The loader will automatically manage this file, effectively keeping it in sync withcue.mod/module.cue. More on this belowUsing
cmd/cueto control/list module dependenciesWhilst it would be possible to maintain
cue.mod/module.cue,cue.mod/sum.cueandgo.modby hand, it would be an incredibly frustrating and error-prone process. Instead,cmd/cuewill be modified to support controlling and listing module dependencies, and updating the files that declare those dependencies.For example, in the course of developing
example.com/blah/v2we might well have run:This resolves
acme.com/quoteto the latest version of a module that provides the packageacme.com/quoteand modifiescue.mod/module.cue,cue.mod/sum.cueandgo.modaccordingly to record the dependency.cue getis therefore directly analogous togo getin terms of its role in dependency management.cue getwill, likecmd/go, support various version queries to control the dependencies being added, removed, upgraded or downgraded. For example@latestis the version query implied when, as above, no@$versionis specified.The
cue listcommand, directly analogous togo list, will be added to provide information about CUE modules and packages. For example we could request information about theacme.com/quotepackage in JSON format as follows:The
cue modcommand will be expanded with subcommands for more fine-grained control ofcue.mod/module.cue,cue.mod/sum.cue(andgo.modindirectly). For example:would add a
replace"directive" tocue.mod/module.cue, which tells the loader to load all versions ofacme.com/quotefrom the directory/path/to/acme.com/quote. The resultingcue.mod/module.cueandgo.modwould then look like this:The
--modflag will, for allcmd/cuecommands that load/resolve package import paths, control the behaviour of resolution. For examplecue list --mod=vendor $pkgwill limit the resolution of the package pattern$pkgto the contents of thecue.mod/pkgdirectory, matching the behaviour of today. The default will be--mod=readonly, withcue getbeing the exception because its whole purpose is to modify the main module's dependencies.Using
cue/loadand thecuelang.org/go/...APIThe Go project entirely abstracts the loading of Go package and module information within
cmd/go.go/packagesexists as a wrapper forcmd/goto load Go packages for inspection and analysis.CUE has a dedicated package for loading CUE instances,
cue/load. This package is used bycmd/cueand users of thecuelang.org/go/...API. As such, there is no need for ago/packagesequivalent, at least not equivalent functionality that wrapscmd/cue.Users of the
cue/loadpackage will benefit from seamless module support, with thecue/load.Configtype being expanded with options relevant to controlling module resolution. The following example demonstrates how a custom proxy serving private modules would be set during the load process, with a specification that the sumdb should not be consulted for those private import paths:Use of a trusted module mirror and checksum database
In August 2019, the Go team at Google launched the Go module mirror and checksum database were launched. As part of the Go 1.13 launch,
cmd/goused both by default. There are some important points to note about this setup:The use of a checksum database is a key component of verified and verifiable builds, or evaluations in CUE terms, and hence trusted reproducible builds (evaluations).
As an interim measure, we propose using proxy.golang.org as a module mirror, and the checksum database sum.golang.org for authentication, until such time as the CUE project can host such services itself. Use of these services will be enabled by default for
cmd/cueandcue/load, but entirely configurable, just likecmd/go. We have permission from the Go team for this proposed use of these services.As covered above, this places one noticeable constraint on the CUE module implementation: that we must include a
go.modfile at the root of a CUE module. This is required for our use of proxy.golang.org and sum.golang.org to satisfy theGOPROXYand sumdb protocols, but also crucially to indicate to the proxy (which runscmd/go) that a repository is Go module-aware, as opposed to not (remember, we don't have such a consideration in CUE). This last point is significant in the case of multi-module repositories, which we intend to support in CUE.Use of the proxy and checksum database will largely fade into the background for users of
cmd/cueandcue/load.For example, under a default configuration and assuming that
acme.com/quoteis public, the following command:would:
acme.com/quote(which we know to be the module providing the package at the same import path);acme.com/quoteas the MVS algorithm proceedsUsing
cue envto understand the currentcmd/cueconfigurationThe
cue envcommand will be added to show the current configuration to the user, for example:Much like
cmd/go, the setting of an explicit environment variable will have the highest precedence, else if a variable is not set then a configuration default can be set in the file located atcue env CUEENVusing thecue env -wcommand. See thecmd/goenvironment variable documentation for more details.Detailed design
Finding a module for a module path
The details in this section largely follow the pattern established with Go module and package resolution.
When the loader needs to resolve an import path to a package and/or module (the concept of a module proxy is fully introduced below), they start by locating the repository that contains the module.
If the module path has a VCS qualifier (one of
.bzr,.fossil,.git,.hg,.svn) at the end of a path component, the loader will use everything up to that path qualifier as the repository URL. For example, for the moduleexample.com/foo.git/bar, the loader will download the repository atexample.com/foo.gitusinggit, expecting to find the module in thebarsubdirectory. The loader will guess the protocol to use based on the protocols supported by the version control tool.If the module path does not have a qualifier, the loader sends an HTTP GET request to a URL derived from the module path with a
?cue-get=1query string. For example, for the moduleacme.com/quote, the loader will send the following request:The loader follows redirects but otherwise ignores response status codes, so the server may respond with a 404 or any other error status. CUE will not support an equivalent of
GOINSECURE.The server must respond with an HTML document containing a
<meta>tag in the document's<head>. The<meta>tag should appear early in the document to avoid confusing the loader's restricted parser. In particular, it should appear before any raw JavaScript or CSS. The<meta>tag must have the form:root-pathis the repository root path, the portion of the module path that corresponds to the repository's root directory. It must be a prefix or an exact match of the requested module path. If it's not an exact match, another request is made for the prefix to verify the<meta>tags match.vcsis the version control system. It must be one ofbzr,fossil,git,hg,svn,mod. Themodscheme instructs the loader to download the module from the given URL using the CUEPROXY protocol (see later). This allows developers to distribute modules without exposing source repositories.repo-urlis the repository's URL. If the URL does not include a scheme (either because the module path has a VCS qualifier or because the<meta>tag lacks a scheme), the loader will try each protocol supported by the version control system. For example, with Git, the loader will tryhttps://thengit+ssh://. Insecure protocols (like http:// and git://) are not supported.As an example, consider
acme.com/quoteagain. The loader sends a request to https://acme.com/quote?cue-get=1. The server responds with an HTML document containing the tag:From this response, the loader will use the Git repository at the remote URL https://github.com/acme.com/quote.
As the CUE modules implementation is based heavily on the Go modules implementation, if a server fails to respond with an appropriate
<meta>tag using the query?cue-get=1, the loader will then attempt to fallback via a query?go-get=1. This means that any server that needs to distinguish the hosting information for Go and CUE modules can do so, whilst not placing an undue burden on existing infrastructure to add support for?cue-get=1queries from day one (the assumption being that generally speaking hosting of CUE and Go modules will generally be aligned in terms of VCS systems). For example, GitHub and other popular hosting services respond to?go-get=1queries for all repositories, so no server reconfiguration is necessary for CUE modules hosted at those sites. Over time it is envisaged that such code hosting sites would add support for?cue-get=1queries.After the repository URL is found, the loader will clone the repository into the module cache. In general, the loader tries to avoid fetching unneeded data from a repository. However, the actual commands used vary by version control system and may change over time. For Git, the loader can list most available versions without downloading commits. It will usually fetch commits without downloading ancestor commits, but doing so is sometimes necessary.
Versions of modules
Much like Go, a version identifies an immutable snapshot of a module, which may be either a release or a pre-release. Each version starts with the letter
v, followed by a semantic version. See the Go module reference for more detail, and Semantic Versioning 2.0.0 for details on how versions are formatted, interpreted, and compared.Therefore, as we have seen in
cue.mod/module.cueexamples above, a module path and version together form the basis of a declared dependency.Module authors explicitly release new versions by defining a semantic version tag within the repository that hosts the module, a tag that indicates which revision should be checked out for that version. For example, as the author of the
example.com/blah/v2module, we would, in a local clone of the repository behindexample.com/blah/v2, do something like:Another module looking to depend on
example.com/blah/v2would then be able to run:and that would resolve to
v2.0.1, specifically the revision 7c5d28e.CUE modules also adopt the concept of a pseudo-version. A pseudo-version is a specially formatted pre-release version that encodes information about a specific revision in a version control repository. For example,
v0.0.0-20191109021931-daa7c04131f5is a pseudo-version. Pseudo versions are used when canonical semantic tagged versions are not available, for example the user wanting to depend on a recent fix pushed to themainbranch of a project. Pseudo-versions follow exactly the same model as implemented for Go modules: see the explanation of pseudo versions and details of how pseudo versions map to commits for more specific information.Declaring dependencies on other modules
The earlier example of the main module
example.com/blah/v2showed a sketch of the schema ofcue.mod/module.cue. This is now presented more fully:With the obvious exception that the representation is different (in CUE a module is defined in CUE itself, whereas in Go
go.modfiles have their own syntax), each of the directives supported ingo.modfiles have corresponding fields and meanings in acue.mod/module.cuefile:moduledirectivecuedirectiverequiredirectiveexcludedirectivereplacedirectiveretractdirectiveBuilding on the short descriptions in the schema above, the links above cover in more detail what each means and when they should be used.
To support the
retractdirective we will provide a builtinsemverpackage that allows for the specification of ranges:Similarly, the
cue.mod/sum.cuefile would have the following schema:Question: do we really need/want to have
cue.mod/sum.cuein CUE format? Or would thego.sumformat suffice?The format of the
go.modfile is described in the Go module reference documentation.Module proxy
As discussed above, as an interim measure we propose using proxy.golang.org as a module mirror, and the checksum database sum.golang.org for authentication, until such time as the CUE project can host such services itself. Use of these services will be enabled by default in the loader. The following CUE environment variables will control use of the proxy and checksum database (this follows almost identically from the Go module environment variables):
CUENOPROXYComma-separated list of glob patterns (in the syntax of Go's
path.Match) of module path prefixes that should always be fetched directly from version control repositories, not from module proxies.If
CUENOPROXYis not set, it defaults toCUEPRIVATE.CUENOSUMDBComma-separated list of glob patterns (in the syntax of Go's
path.Match) of module path prefixes for which the loader should not verify checksums using the checksum database.If
CUENOSUMDBis not set, it defaults toCUEPRIVATE.CUEPRIVATEComma-separated list of glob patterns (in the syntax of Go's
path.Match) of module path prefixes that should be considered private.CUEPRIVATEis a default value forCUENOPROXYandCUENOSUMDB.CUEPRIVATEalso determines whether a module is considered private forCUEVCS(see below).CUEPROXYList of module proxy URLs, separated by commas (
,) or pipes (|). When the loader looks up information about a module, it contacts each proxy in the list in sequence until it receives a successful response or a terminal error. A proxy may respond with a 404 (Not Found) or 410 (Gone) status to indicate the module is not available on that server.The loader's error fallback behaviour is determined by the separator characters between URLs. If a proxy URL is followed by a comma, the loader falls back to the next URL after a 404 or 410 error; all other errors are considered terminal. If the proxy URL is followed by a pipe, the loader falls back to the next source after any error, including non-HTTP errors like timeouts.
CUEPROXYURLs may have the schemeshttpsorfile. If a URL has no scheme,httpsis assumed. A module cache may be used directly as a file proxy:Two keywords may be used in place of proxy URLs:
off: disallows downloading modules from any source.direct: download directly from version control repositories instead of using a module proxy.CUEPROXYdefaults tohttps://proxy.golang.org,direct. Under that configuration, the loader first contacts the Go module mirror run by Google, then falls back to a direct connection if the mirror does not have the module. See https://proxy.golang.org/privacy for the mirror's privacy policy. TheCYEPRIVATEandCUENOPROXYenvironment variables may be set to prevent specific modules from being downloaded using proxies.CUESUMDBIdentifies the name of the checksum database to use and optionally its public key and URL. For example:
The loader knows the public key of
sum.golang.organd also that the namesum.golang.google.cn(available inside mainland China) connects to thesum.golang.orgdatabase; use of any other database requires giving the public key explicitly. The URL defaults tohttps://followed by the database name.CUESUMDBdefaults tosum.golang.org, the Go checksum database run by Google. See https://sum.golang.org/privacy for the service's privacy policy.If
CUESUMDBis set tooffthe checksum database is not consulted, and all unrecognised modules are accepted, at the cost of giving up the security guarantee of verified repeatable downloads for all modules. A better way to bypass the checksum database for specific modules is to use theCUEPRIVATEorCUENOSUMDBenvironment variables.Module versioning
(This follows directly from the Go module reference
Starting with major version 2, module paths must have a major version suffix like
/v2that matches the major version. For example, if a module has the pathexample.com/modatv1.0.0, it must have the pathexample.com/mod/v2at versionv2.0.0.Major version suffixes are not allowed at major versions
v0orv1. There is no need to change the module path betweenv0andv1becausev0versions are unstable and have no compatibility guarantee. Additionally, for most modules,v1is backwards compatible with the lastv0version; av1version acts as a commitment to compatibility, rather than an indication of incompatible changes compared withv0.Minimal version selection
It is proposed that, like Go, minimal version selection (MVS) will be used as the algorithm to select a set of module versions to use when evaluating packages. MVS is described in detail in Minimal Version Selection by Russ Cox. The detail of the algorithm is not covered here.
MVS operates on a directed graph of modules, specified with go.mod files. Each vertex in the graph represents a module version. Each edge represents a minimum required version of a dependency, specified using a require directive. replace and exclude directives in the main module's go.mod file modify the graph.
Resolving a package to a module
When the loader loads a package using a package path, it needs to determine which module provides the package. CUE will follow exactly the same model for resolving a package to a module in this respect, with the obvious substitution replacing "the
gocommand" with "the loader", andGO*withCUE*environment module-related environment variables. Whilst CUE does not have the equivalent ofGOOSorGOARCH, it will similarly ignore file level build constraints of the form@ifduring this resolution.Changes to
cmd/cueThis section introduces changes that will be made to
cmd/cue. Most of these changes are effectively a front-end to changes that will be made tocue/load; those changes are discussed in the next section.All
cmd/cuecommands that load information about packages will become module-aware:cue cmdcue defcue evalcue exportcue fixcue fmtcue getcue importcue trimcue vetThe
--modflag is understood by module-aware commands and controls the resolution of packages in the following way (likecmd/go):--mod=modtells the loader to ignore thecue.mod/pkgvendor directory and to automatically updatecue.mod/module.cue(andgo.mod), for example, when an imported package is not provided by any known module.--mod=readonlytells the loader to ignore thecue.mod/pkgvendor directory and to report an error ifcue.mod/module.cue(orgo.mod) needs to be updated.--mod=vendortells the loader to use thecue.mod/pkgvendor directory. In this mode, the loader will not use the network or the module cache.By default, if a
cue.mod/pkgvendor directory is present at the module root, the loader acts as if--mod=vendorwere used. Otherwise, the loader acts as if--mod=readonlywere used.Module-aware commands will also understand
--modpathas a means of specifying an alternative path at which acue.moddirectory can be found (and correspondingly readcue.mod/module.cueand associated files from).Like
cmd/go, the--modcacherwflag instructs the loader to create new directories in the module cache with read-write permissions instead of making them read-only.We now move on to talk in more details about changes to
cmd/cuecommands. For users of thecuelang.org/go/...APIs, programmatically creating and running command instances viacuelang.org/go/cmd/cue/cmdwill remain possible and will be fully module-aware.For any command that talks about modifying
cue.mod/module.cue, it should be assumed an identical change will be made to the modulego.modfile, unless specified otherwise. The same generally applies for any action of the loader that would modify any of these files.Much of the proposal regarding
cmd/cuecommands is, unsurprisingly, heavily based on thecmd/go-equivalent commands.cue getcue getupdates module dependencies in thecue.mod/module.cuefile for the main module.Unlike
cmd/gowe have no need to establish the-dflag because there is no concept of building/installing the module/package we have just fetched. Nor do we have a need for-tuntilcue test. Otherwise the command will behave likego get:cue getaccepts a list of packages, package patterns, and module paths as arguments.cue getupdates the module that provides the package.allor a path with a...wildcard),cue getexpands the pattern to a set of packages, then updates the modules that provide the packagesacme.com/nopkghas no package in its root directory),cue getwill update the modulecue get acme.com/quote@v1.1.0. A version query suffix consists of an@symbol followed by a version query, which may indicate a specific version (v1.1.0), a version prefix (v1.1), a branch or tag name (main), a revision (1234abcd), or one of the special querieslatest,upgrade,patch, ornone. If no version is given,cue getuses the@upgradequery.@none. This is a special kind of downgrade. Modules that depend on the removed module will be downgraded or removed as needed. A module requirement may be removed even if one or more of its packages are imported by packages in the main module. In this case, the next build command may add a new module requirement.Once
cue gethas resolved its arguments to specific modules and versions,cue getwill add, change, or remove require directives in the main module'scue.mod/module.cuefile to ensure the modules remain at the desired versions in the future. Note that required versions incue.mod/module.cuefiles are minimum versions and may be increased automatically as new dependencies are added. See Minimal version selection (MVS) for details on how versions are selected and conflicts are resolved by module-aware commands.cue getthen proceeds along the lines described in thego getdocumentation.cue listcue listlists the named packages, one per line. The most commonly-used flags are-fand-json, which control the form of the output printed for each package. Other list flags, documented below, control more specific details.The default output shows the package import path:
The
--fflag specifies an alternate format for the list, using the syntax of packagetext/template. The default output is equivalent to--f '{{.ImportPath}}'. The struct being passed to the template is:With error information defined as:
The
Modulestruct type is defined as below forcue list -m.cue list -mcue list -mlists information about CUE modules. The--mflag lists information about modules and not packages.The
--jsonflag prints JSON-encoded output according to the struct type:As an alternative to
--json, the--fflag specifies--uadds information about available upgradesThe
--versionsflag causes list to set the module's Versions field to a list of all known versions of that module, ordered according to semantic versioning, lowest to highest.--retractedflag instructs list to show retracted versions in the list printed with the -versions flag and to consider retracted versions when resolving version queriescue mod downloadThe
cue mod downloadcommand downloads the named modules into the module cache. Arguments can be module paths or module patterns selecting dependencies of the main module or version queries of the formpath@version. With no arguments, download applies to all dependencies of the main module.The loader will automatically download modules as needed during ordinary execution. The
cue mod downloadcommand is useful mainly for pre-filling the module cache or for loading data to be served by a module proxy.By default,
downloadwrites nothing to standard output. It prints progress messages and errors to standard error.The
--jsonflag causes download to print a sequence of JSON objects to standard output, describing each downloaded module (or failure), corresponding to this Go struct:The
--xflag causes download to print the commands download executes to standard error.cue mod editExample:
The
cue mod editcommand provides a command-line interface for editing and formattingcue.mod/module.cuefiles, for use primarily by tools and scripts.cue mod editreads only onecue.mod/module.cuefile; it does not look up information about other modules. By default,cue mod editreads and writes thecue.mod/module.cuefile of the main module, but a different target file can be specified after the editing flags. All changes tocue.mod/module.cuefiles made viacue mod editwill be reflected in a CUE module'sgo.modfile.The editing flags specify a sequence of editing operations.
--moduleflag changes the module's path (thecue.mod/modue.cuefile's module directive).--cue=versionflag sets the expected Go language version.--require=path@versionand--droprequire=pathflags add and drop a requirement on the given module path and version. Note that--requireoverrides any existing requirements onpath. These flags are mainly for tools that understand the module graph. Users should prefercue get path@versionorcue get path@none, which make othergo.modadjustments as needed to satisfy constraints imposed by other modules--exclude=path@versionand--dropexclude=path@versionflags add and drop an exclusion for the given module path and version. Note that--exclude=path@versionis a no-op if that exclusion already exists.--replace=old[@v]=new[@v]flag adds a replacement of the given module path and version pair. If the@vinold@vis omitted, a replacement without a version on the left side is added, which applies to all versions of the old module path. If the@vinnew@vis omitted, the new path should be a local module root directory, not a module path. Note that--replaceoverrides any redundant replacements forold[@v], so omitting@vwill drop replacements for specific versions.--dropreplace=old[@v]flag drops a replacement of the given module path and version pair. If the@vis provided, a replacement with the given version is dropped. An existing replacement without a version on the left side may still replace the module. If the@vis omitted, a replacement without a version is dropped.--retract=versionand--dropretract=versionflags add and drop a retraction for the given version, which may be a single version (likev1.2.3) or an interval (like[v1.1.0,v1.2.0]). Note that the--retractflag cannot add a rationale comment for theretractdirective. Rationale comments are recommended and may be shown bycue list -m -uand other commands.The editing flags may be repeated. The changes are applied in the order given.
cue mod graphThe
cue mod graphcommand prints the module requirement graph (with replacements applied) in text form. For example:Each vertex in the module graph represents a specific version of a module. Each edge in the graph represents a requirement on a minimum version of a dependency.
cue mod graphprints the edges of the graph, one per line. Each line has two space-separated fields: a module version and one of its dependencies. Each module version is identified as a string of the formpath@version. The main module has no@versionsuffix, since it has no version.See Minimal version selection (MVS) for more information on how versions are chosen. See also
cue list -mfor printing selected versions andcue mod whyfor understanding why a module is needed.cue mod initThe
cue mod initcommand initialises and writes a newcue.mod/module.cuefile in the current directory, in effect creating a new module rooted at the current directory. Thecue.moddirectory must not already exist. Ago.modfile will also be written to the current directory.Per the current module docs, the use of a module is optional, but required if one wants to import files. The module name is required if a package within the module needs to import another package within the main module.
cue mod tidycue mod tidyensures that thecue.mod/module.cuefile (and by extension thego.modfile) matches the source code in the module. It adds any missing module requirements necessary to build the current module's packages and dependencies, and it removes requirements on modules that don't provide any relevant packages. It also adds any missing entries tocue.mod/sum.cueand removes unnecessary entries.The
-eflag causescue mod tidyto attempt to proceed despite errors encountered while loading packages.The
-vflag causescue mod tidyto print information about removed modules to standard error.cue mod tidyworks by loading all of the packages in the main module and all of the packages they import, recursively.cue mod tidyacts as if all build constraints are enabled, so it will consider@ifconstrained files even if those source files wouldn't normally be evaluated.TODO: do we need the equivalent of the
ignorebuild constraint exception?Like the
...package pattern,cue mod tidywill not consider packages in the main module in directories namedtestdataor with names that start with.or_unless those packages are explicitly imported by other packages.Once
cue mod tidyhas loaded this set of packages, it ensures that each module that provides one or more packages either has arequiredirective in the main module'scue.mod/module.cuefile or is required by another required module.cue mod tidywill add a requirement on the@latestversion on each missing module.cue mod tidywill removerequiredirectives for modules that don't provide any packages in the set described above.cue mod tidymay also add or removeindirectfields on#Requiredirectives. A#Requiredirective withindirect: truedenotes a module that does not provide packages imported by packages in the main module. These requirements will be present if the module that imports packages in the indirect dependency has an incompletecue.mod/module.cuefile. They may also be present if the indirect dependency is required at a higher version than is implied by the module graph; this usually happens after running a command likecue get -u ./....cue mod vendorThe
cue mod vendorcommand constructs a directory namedcue.mod/pkgthat contains copies of all packages needed to support evaluations of packages in the main module. As withcue mod tidyand other module commands, build constraints (except forignore???) are not considered when constructing thecue.mod/pkgvendor directory.When vendoring is enabled, the loader will load packages from the
cue.mod/pkgvendor directory instead of downloading modules from their sources into the module cache and using packages those downloaded copies.cue mod vendoralso creates the filevendor/modules.txtthat contains a list of vendored packages and the module versions they were copied from. When vendoring is enabled, this manifest is used as a source of module version information. When thecuecommand readsvendor/modules.txt, it checks that the module versions are consistent withcue.mod/module.cue(andgo.mod). If eithercue.mod/module.cueorgo.modchanged sincevendor/modules.txtwas generated,cue mod vendorshould be run again.Note that
cue mod vendorremoves thecue.mod/pkgvendor directory if it exists before re-constructing it. Local changes should not be made to vendored packages. Thecuecommand does not check that packages in thecue.mod/pkgvendor directory have not been modified, but one can verify the integrity of thecue.mod/pkgvendor directory by runningcue mod vendorand checking that no changes were made.The
--eflag causescue mod vendorto attempt to proceed despite errors encountered while loading packages.The
--vflag causescue mod vendorto print the names of vendored modules and packages to standard error.cue mod verifycue mod verifychecks that dependencies of the main module stored in the module cache have not been modified since they were downloaded. To perform this check,cue mod verifyhashes each downloaded module.zipfile and extracted directory, then compares those hashes with a hash recorded when the module was first downloaded.cue mod verifychecks each module in the evaluation list (which may be printed withcue list -m all.If all the modules are unmodified,
cue mod verifyprints "all modules verified". Otherwise, it reports which modules have been changed and exits with a non-zero status.Note that all module-aware commands verify that hashes in the main module's
cue.mod/sum.cuefile match hashes recorded for modules downloaded into the module cache. If a hash is missing fromcue.mod/sum.cue(for example, because the module is being used for the first time), the loader verifies its hash using the checksum database (unless the module path is matched byCUEPRIVATEorCUENOSUMDB).In contrast,
cue mod verifychecks that module.zipfiles and their extracted directories have hashes that match hashes recorded in the module cache when they were first downloaded. This is useful for detecting changes to files in the module cache after a module has been downloaded and verified.cue mod verifydoes not download content for modules not in the cache, and it does not usecue.mod/sum.cuefiles to verify module content. However,cue mod verifymay downloadgo.modfiles in order to perform minimal version selection. It will usecue.mod/sum.cueto verify those files, and it may addcue.mod/sum.cueentries for missing hashes.cue mod whycue mod whyshows a shortest path in the import graph from the main module to each of the listed packages.The output is a sequence of stanzas, one for each package or module named on the command line, separated by blank lines. Each stanza begins with a comment line starting with
#giving the target package or module. Subsequent lines give a path through the import graph, one package per line. If the package or module is not referenced from the main module, the stanza will display a single parenthesised note indicating that fact.The
--mflag causescue mod whyto treat its arguments as a list of modules.cue mod whywill print a path to any package in each of the modules. Note that even when--mis used,cue mod whyqueries the package graph, not the module graph printed bycue mod graph.By default,
cue mod whyconsiders the graph of packages matched by theallpattern, which is the same set of packages matched bygo mod vendor.cue clean -modcacheThe
--modcacheflag causescue cleanto remove the entire module cache, including unpacked source code of versioned dependencies.This is usually the best way to remove the module cache. By default, most files and directories in the module cache are read-only to prevent tests and editors from unintentionally changing files after they've been authenticated. Unfortunately, this causes commands like
rm -rto fail, since files can't be removed without first making their parent directories writable.The
--modcacherwflag (accepted by module-aware commands) causes new directories in the module cache to be writable. To pass--modcacherwto all module-aware commands, add it to theGOFLAGSvariable.GOFLAGSmay be set in the environment or withcue env -w.--modcacherwshould be used with caution; developers should be careful not to make changes to files in the module cache.cue mod verifymay be used to check that files in the cache match hashes in the main module'scue.mod/sum.cuefile.cue envThis is covered in the "Proposal" section above.
Required changes to
cue/loadThe main changes required in
cue/loadare additions to thecue/load.Configtype. These are backwards compatible assuming that users of this type are, as advised bygo vet, using keyed struct literals. All field additions below have corresponding "front end" flags/environment variables incmd/cue.As is the case today,
cmd/cuewill usecue/loadto load CUE instances, defaulting the values of modules-relatedcue/load.Configvalues from flags and environment variables (see "Environment variables").For users of
cue/loadwho want to mimic the behaviour ofcmd/cue, a utility function that sets the modules-related fields ofcue/load.Configtocmd/cuedefaults (according to the values of environment variables and defaults) will be provided.Version queries
We re-use the same concept of version queries as Go: https://golang.org/ref/mod#version-queries
A version query may be one of the following:
v1.2.3, which selects a specific version. See Versions for syntax.v1orv1.2, which selects the highest available version with that prefix.<v1.2.3or>=v1.5.6, which selects the nearest available version to the comparison target (the lowest version for>and>=, and the highest version for<and<=).v2selects the latest version starting withv2, not the branch namedv2.latest, which selects the highest available release version. If there are no release versions,latestselects the highest pre-release version. If there no tagged versions,latestselects a pseudo-version for the commit at the tip of the repository's default branch.upgrade, which is likelatestexcept that if the module is currently required at a higher version than the versionlatestwould select (for example, a pre-release),upgradewill select the current version.patch, which selects the latest available version with the same major and minor version numbers as the currently required version. If no version is currently required,patchis equivalent tolatest. Since Go 1.16,go getrequires a current version when usingpatch(but the-u=patchflag does not have this requirement).Release versions are preferred over pre-release versions. For example, if versions v1.2.2 and v1.2.3-pre are available, the latest query will select v1.2.2, even though v1.2.3-pre is higher. The <v1.2.4 query would also select v1.2.2, even though v1.2.3-pre is closer to v1.2.4. If no release or pre-release version is available, the latest, upgrade, and patch queries will select a pseudo-version for the commit at the tip of the repository's default branch. Other queries will report an error.
cmd/cueoutside of a module contextLike today,
cmd/cuewill continue operate outside of a module context, and only fail if its arguments require resolution of non-builtins.GOPROXYprotocolCUE module proxies will implement the
GOPROXYprotocol. Anyone looking to provide a CUE module proxy alternative to proxy.golang.org should consult theGOPROXYreference.Version control systems
The loader may download module source code and metadata directly from a version control repository. Downloading a module from a proxy is usually faster, but connecting directly to a repository is necessary if a proxy is not available or if a module's repository is not accessible to a proxy (frequently true for private repositories). Git, Subversion, Mercurial, Bazaar, and Fossil are supported. A version control tool must be installed in a directory in
PATHin order for the loader to use it.To download specific modules from source repositories instead of a proxy, set the
CUEPRIVATEorCUENOPROXYenvironment variables (or equivalent options incue/load.Config). To configure the loader to download all modules directly from source repositories, setCUEPROXYtodirect. See Environment variables for more information.See https://golang.org/ref/mod#vcs for more details on the specifics of:
go-import<meta>tagsControlling version control tools with
CUEVCSThe loader's ability to download modules with version control commands like
gitis critical to the decentralized package ecosystem, in which code can be imported from any server. It is also a potential security problem if a malicious server finds a way to cause the invoked version control command to run unintended code.The CUE module implementation will follow the same model of
GOVCSto change the allowed version control systems for specific modules, via the variableCUEVCS. For example:With this setting, code with a module or import path beginning with
github.com/can only usegit; paths onevil.comcannot use any version control command, and all other paths (*matches everything) can use onlygitorhg.See the
GOVCSreference documentation for more details.Module zip files
Like Go, CUE module versions are distributed as
.zipfiles. There is rarely any need to interact directly with these files, since the loader creates, downloads, and extracts them automatically from module proxies and version control repositories. However, it's still useful to know about these files to understand cross-platform compatibility constraints or when implementing a module proxy.The
cue mod downloadcommand downloads zip files for one or more modules, then extracts those files into the module cache. Depending onCUEPROXYand other environment variables, the loader may either download zip files from a proxy or clone source control repositories and create zip files from them. The--jsonflag may be used to find the location of download zip files and their extracted contents in the module cache.CUE modules will be subject to the same constraints as Go modules with respect to file path and size constraints. See https://golang.org/ref/mod#zip-files for more details
Private modules
We establish the same support model for private modules as Go modules, with the following environment variables as substitutions:
CUEPROXY— list of module proxy URLs. Thegocommand will attempt to download modules from each server in sequence. The keyworddirectinstructs thegocommand to download modules from version control repositories where they're developed instead of using a proxy.CUEPRIVATE— list of glob patterns of module path prefixes that should be considered private. Acts as a default value forCUENOPROXYandCUENOSUMDB.CUENOPROXY— list of glob patterns of module path prefixes that should not be downloaded from a proxy. Thegocommand will download matching modules from version control repositories where they're developed, regardless ofCUEPROXY.CUENOSUMDB— list of glob patterns of module path prefixes that should not be checked using the public checksum database, sum.golang.org.CUEINSECURE— list of glob patterns of module path prefixes that may be retrieved over HTTP and other insecure protocols.The Go module documentation provides a complete set of scenarios covering the various permutations of public/private modules:
It also provides a comprehensive explanation of how the loader (
cmd/goin the case of the Go modules implementation) handles privacy concerns with respect to proxy requests. See the Go modules Privacy section for a comprehensive explanation.Modules cache
Like Go, CUE will establish and use a user module cache, a directory where the loader stores downloaded module files. The default location of the module cache is
$HOME/cue/modcache. To use a different location, set theCUEMODCACHEenvironment variable.The cache may be shared by multiple CUE projects developed on the same machine. The loader will use the same cache regardless of the location of the main module. Multiple instances of the loader may safely access the same module cache at the same time.
For more detail on the module cache implementation, see the Go module cache reference.
Authenticating modules
The approach and implementation of authenticating modules follows exactly from the
cmd/goimplementation, and uses the checksum databse sum.golang.org by default for public modules.When the loader downloads a module zip file or
go.modfile (which is why this file is necessary for compatibility with the Go proxy and checksum models) into the module cache, it computes a cryptographic hash and compares it with a known value to verify the file hasn't changed since it was first downloaded. The loader reports a security error if a downloaded file does not have the correct hash.For
go.modfiles, the loader computes the hash from the file content. For module zip files, the loader computes the hash from the names and contents of files within the archive in a deterministic order. The hash is not affected by file order, compression, alignment, and other metadata. Seegolang.org/x/mod/sumdb/dirhashfor hash implementation details.The loader compares each hash with the corresponding entry in the main module's
cue.mod/sum.cuefile. If the hash is different from the hash incue.mod/sum.cue, the loader reports a security error and deletes the downloaded file without adding it into the module cache.If the
cue.mod/sum.cuefile is not present, or if it doesn't contain a hash for the downloaded file, the loader may verify the hash using the checksum database, a global source of hashes for publicly available modules. Once the hash is verified, the loader adds it tocue.mod/sum.cueand adds the downloaded file in the module cache. If a module is private (matched by theCUEPRIVATEorCUENOSUMDBenvironment variables) or if the checksum database is disabled (by settingCUESUMDB=off), the loader accepts the hash and adds the file to the module cache without verifying it.The module cache is usually shared by all CUE projects on a system, and each module may have its own
cue.mod/sum.cuefile with potentially different hashes. To avoid the need to trust other modules, the loader verifies hashes using the main module'scue.mod/sum.cuewhenever it accesses a file in the module cache. Zip file hashes are expensive to compute, so the loader checks pre-computed hashes stored alongside zip files instead of re-hashing the files. Thecue mod verifycommand may be used to check that zip files and extracted directories have not been modified since they were added to the module cache.The format of
cue.mod/sum.cuefiles is described above, and follows directly from thego.sumstructure (albeit a different format). For details on thego.sumfor and checksum databases, see the corresponding sections in the Go modules reference.The
cue.moddirectoryCurrently (the CUE world prior to this proposal) the contents of the
cue.moddirectory have the following function/semantics:Given a non-main module import path
acme.com/quote, the loader unifies the contents of the package valuescue.mod/{gen,pkg,usr}/acme.com/quote.Under this proposal, the only change to these semantics is that
cue.mod/pkgbecomes the equivalent of Go modules'vendordirectory:If
cue.mod/pkgexists, then the loader will expect to be able to load all non-main module imports from there - this also retains compatibility for the existing loading mechanism, i.e. when nogo.modfile exists in the CUE module root. Given a non-main module import pathacme.com/quote, the loader will unify the contents of the package valuescue.mod/{gen,pkg,usr}/acme.com/quoteas it does today.If
cue.mod/pkgdoes not exist, then the loader will resolve and load package dependencies from the module cache. Given a non-main module import pathacme.com/quote, the loader unifies the contents of the package valuescue.mod/{gen,usr}/acme.com/quotewith, for example,$(cue env CUEMODCACHE)/acme.com/quote@v1.1.0.Environment variables
Explanation of the flags and environment variables (and config options in the case of
cue/load) that control the loader's behaviour are covered elsewhere in this proposal. A full list is provided here for reference:--mod- controls whether the loader can automatically updatecue.mod/module.cueand associated files, or usecue.mod/pkg--modpath- an alternative path at which acue.moddirectory can be found (and correspondingly readcue.mod/module.cueand associated files from)CUEMODCACHE- the directory where the loader will store downloaded modules and related filesCUENOPROXY- Comma-separated list of glob patterns (in the syntax of Go'spath.Match) of module path prefixes that should always be fetched directly from version control repositories, not from module proxies. IfCUENOPROXYis not set, it defaults toCUEPRIVATE.CUENOSUMDB- Comma-separated list of glob patterns (in the syntax of Go'spath.Match) of module path prefixes for which the go should not verify checksums using the checksum database. IfCUENOSUMDBis not set, it defaults toCUEPRIVATE.CUEPRIVATE- Comma-separated list of glob patterns (in the syntax of Go'spath.Matchof module path prefixes that should be considered private.CUEPRIVATEis a default value forCUENOPROXYandCUENOSUMDB.CUEPRIVATEalso determines whether a module is considered private forCUEVCS.CUEPROXY- List of module proxy URLs, separated by commas (,) or pipes (|).CUESUMDB- Identifies the name of the checksum database to use and optionally its public key and URLCUEVCS- Controls the set of version control tools thegocommand may use to download public and private modules (defined by whether their paths match a pattern inCUEPRIVATE) or other modules matching a glob pattern.The mapping from these flags and environment variables to
cue/load.Configoptions is covered in "Required changes tocue/load".Go modules and CUE modules coexisting
It is not unreasonable to imagine a CUE module and Go modules co-existing at the same path, sharing the same VCS repository. Indeed, with the native support for exporting CUE to Go, and importing CUE from Go, it seems very likely that these situations will arise. This proposal supports such a setup.
The versioning of both modules would be intrinsically linked by virtue of each module system sharing the same tagging scheme in the same repository. However, assuming a Go module and CUE module exist at the same root, it would not be possible to version the two separately within the same repository. On the assumption that co-existence of CUE and Go code implies a strong relationship between the two, with breaking changes in one almost certainly corresponding to breaking changes in the other, we don't foresee this being a problem.
Versioning the two modules separately would still be possible, but only in separate VCS repositories. This would be achieved by having the
go-importandcue-importmeta tags return different repository locations. See "Finding a module for a module path" for more information.Where a Go and CUE module coexist in the same repository, there would be some redundancy insofar as a CUE module would contain Go code, and vice versa. We don't foresee any specific problems beyond the limited inefficiency in CPU, memory and storage terms of this redundancy.
The post-
go.modfutureAs indicated in the summary of this proposal, use of proxy.golang.org and sum.golang.org is intended as an interim measure until such time as the CUE project can host such services itself. The requirement for a
go.modfile in a CUE module is tied to our use of those services.When the CUE project can host such services itself, we will need to develop a
CUEPROXYprotocol, similar to theGOPROXYprotocol, and hosting a service that speaks that protocol. We would then look to create a number of CUE releases wherecmd/cueknows how to speak this protocol, at which point projects would be able to start the process of removinggo.modfiles from their CUE modules. The only challenge here being that a project would only be able to remove ago.modif it could be sure there are no consumers who rely on using acmd/cueversion that does not speak the new protocol. However, adopting something akin to the Go release policy would be sensible in this respect: we would only advisego.modfiles be removed when there are two major versions of CUE that support the newCUEPROXYprotocol.However, in the context of the previous section - "Go modules and CUE modules co-existing" - splitting out CUE dependencies from Go dependencies does appear to create an issue. Consider the following example, in a post-
go.modworld, where CUE dependencies are not added to thego.modfile, and ago.modfile is not required as part of a CUE module:(we elide
go.sumandcue.mod/sum.cuefiles for simplicity).Notes:
go.modfile exists because we have also declared a Go module; hence we have two main modules, one Go, one CUE, and both coincide (have the same module directory root)go.modworld, thego.modfile only reflects the dependencies of the Go module; the CUE dependencies are reflected incue.mod/module.cueacme.com/quoteis both the path of a Go module and a CUE modulev1.2.0in the source code repository that contains both creates versionsv1.2.0of bothacme.com/quoterequired by the Go module (v1.2.0) is different that required by the CUE module (v1.1.0)The problem is that the version of
acme.com/quoteresolved ingo.modis not guaranteed to be the same version ofacme.com/quoteresolved incue.mod/module.cue. We essentially have two different instances of MVS running with different constraints.Much of the time this might be totally innocuous. But version skew like this will almost certainly lead to issues, for example in the case of additive changes to an API.
Nor is this problem unique to the combination of Go and CUE. It is envisaged that languages other than Go will be supported via
cue importandcue export. Each language will have its own (or indeed many) package versioning system equivalent. This scenario of a coincident Language X and CUE main module (or equivalent in Language X's terms) depending on coincident Language X and CUE modules is therefore more widespread: the version resolution algorithm of Language X is not guaranteed to arrive at the same revision as the CUE MVS algorithm.Notable details/exceptions
Here is a general list of points that don't naturally fit under any other heading:
gopkg.in. However, such a position does not preclude doing so in the futureCUE & A
Why have you chosen to use {proxy,sum}.golang.org?
The Go module specification and implementation do not depend upon either proxy.golang.org or sum.golang.org. Indeed the
GOPROXYand checksum protocols provide the necessary abstraction. Therefore the next phase of CUE module support does not need to be tied to either the module mirror at proxy.golang.org or checksum database at sum.golang.org. However, there are some significant advantages to doing so:cmd/go...These seem to outweigh the disadvantages:
go.modfile at the root of each CUE module (which will be done entirely automatically)The most crucial point however is that the default of using proxy.golang.org and sum.golang.org within the loader is just that: a default. It is entirely possible to turn off use of both via
CUEPROXY=offandCUESUMDB=off.What about
cue test?As is covered elsewhere in this proposal, we have not concluded a design for
cue test(this falls under cuelang/cue#209). However, this proposal is entirely compatible with being extended to support_test.cuefiles, again following the pattern of_test.gofiles in Go.What about a
//go:embedCUE equivalent?As of Go 1.16,
cmd/gonow supports including static files and file trees as part of the final executable, using the new//go:embeddirective. See the documentation for the newembedpackage for details.We are working on a proposal for how to support the concept of embedding in CUE. Much like
cue test, this proposal is entirely compatible with (and indeed depends upon) this proposal, the next phase of modules.What about a
goreleaseequivalent?goreleaseis an experimental tool that helps module authors avoid common problems before releasing a new version of a module.Examples:
goreleaseanalyzes changes in the public API and dependencies of the main module. It compares a base version with the currently checked out revision. Given a proposed version to release,goreleasereports whether the changes are consistent with semantic versioning.Given the very nature of the problems that CUE looks to solve, it will be entirely possible to provide such a command to help CUE module authors. Much like
goreleaseis intended to becomego release, the equivalent in the CUE world would likely be spelledcue release.Will CUE packages start to pollute pkg.go.dev?
Whilst we have permission to use proxy.golang.org and sum.golang.org from the Go team until such time as the CUE project starts to host instances of a module mirror and checksum database itself, we should look to limit any unintended side effects. One such side effect would that pkg.go.dev (a Go module and package documentation and discovery site) uses index.golang.org (an index which serves a feed of new module versions that become available at proxy.golang.org). CUE modules would therefore start to "leak" into pkg.go.dev results. We will work with the pkg.go.dev team to ensure that the relevant heuristics for determining CUE-only modules.
How does this proposal relate to
io/fs.FS?Go 1.16 introduce a new
io/fspackage that defines thefs.FSinterface, an abstraction for read-only trees of files. This package largely exists to support the new//go:embedfeature, but does have other uses.cuelang/cue#607 raises the question of how the existing
cue/load.Config.Overlayfield might be used to supply an entire file system as input tocue/load. cuelang/cue#607 (comment) clarifies that the currentOverlayfield exists to complement the the operating system file system, rather than replace it. As outlined in that comment, however, adding anio/fs.FSfield tocue/load.Configwould allow the intended semantics. This modules proposal is fully compatible with this proposal, but necessarily orthogonal to it.What about verifiable evaluations?
One of the main differences between Go and CUE from a "build" perspective is that Go has complete control over build artefacts. Specifically, binaries that represent the compilation result of a
mainpackage.cmd/goincludes sufficient module-related information (module path, version and checksum) in those binary artefacts so as to enable verifiable builds.runtime/debug.ReadBuildInfo()gives runtime access to that information;go version -m /path/to/binaryallows it to be inspected. See Russ Cox's blog post for more information.CUE does not have such control over its many output formats (JSON, Yaml, JSONSchema etc). An immediate consideration here would be that comments be used to encode similar module-related information. However, JSON for one does not support comments.
The Required fields and related issues proposal includes a section on
cue export, and how that would be repurposed to be the inverse ofcue import. The example presented there is as follows.Given the CUE file:
cue exportwould then produce the followingtxtaroutput:One option therefore would be to make a step towards verifiable evaluations by including sufficient module-related information (like that included Go binaries) in the
txtaroutput ofcue export.Are there any alternatives to requiring a
go.mod?If we want to utilise and leverage the existing module mirror and checksum database, we don't see a way around the requirement of declaring a
go.modfile in the root of a CUE module. TheGOPROXYprotocol requires that a regulargo.modfile (i.e. a symlink is not sufficient) denote the root of a Go module, and that that file declares the module's requirements, retractions etc.As noted above however, this is an interim measure until such time as the CUE project can host such services itself.
What about anonymous modules? Will they need a
go.mod?Anonymous modules can be created today via:
This creates a
cue.mod/module.cuefile as follows:Anonymous modules are useful because as an end user, i.e. a situation where you know a package within the module will never be a dependency of another module, coming up with a module name is an annoying problem.
Go modules do not support anonymous modules: every module must have a path. Hence we simply could not maintain a parallel
go.modfile, matching the requirements listed incue.mod/module.cue.Therefore, for anonymous modules,
cmd/cueandcue/loadwill not create or maintain ago.modfile.Why is the module cache in the user's home directory?
Russ Cox provided excellent motivation for this decision in a GitHub discussion about the
GOMODCACHEenvironment variable:What is the timeline for CUE modules?
Once the CUE community has had a chance to consider and respond to this proposal, if there is broad agreement with the direction implementation would start immediately. As mentioned elsewhere in this proposal, we hope to reuse much of the
cmd/go/internal/...implementation, as well as learning from and leveraging the vast experience of the Go team.Coming up with a rough timeline and priority ordered list of work will be the first thing we do when starting work on this next phase of CUE module support.
Are submodules and multi-module repositories supported?
Yes. Although the same advice regarding both will exist in the CUE world. As a starting point the following advice from Russ Cox will likely hold true for the vast majority:
For more details see https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories
What strategies exist for supporting multiple major versions of a module in parallel?
A corollary of the import compatibility rule:
is that any breaking changes in a module with major version number
>=1must be accompanied by an increase in major version number. This raises the question of how to support users of the now old major version - is it possible to support both at the same time?Like with Go modules, developers will have two options when it comes to maintaining multiple major versions of a module in parallel:
See the Go modules wiki entry on the topic as well as a mention in the Go modules reference.
Should the CUE version tags be namespaced?
Some of the early discussions about
vgo(the Go modules prototype) questioned whether Go should distinguish the VCS used tags used to indicated versions of Go modules. The thinking being thatv1.1.0says nothing about the fact the tag corresponds to a version of a Go module. Alternatives included namespacing those tags, e.g.go:v1.1.0.Link to those discussions
It is natural and appropriate to consider the same question in the context of designing CUE package versioning and modules.
However, if we choose to base our implementation on Go modules, using proxy.golang.org and sum.golang.org, then by definition we adopt the same approach to creating module versions: tagging with semantic versions, e.g.
v1.1.0. Tags indicating Go versions and CUE versions will therefore be indistinguishable.But as we cover under "Go modules and CUE modules coexisting", we don't see a problem with this "conflict" - indeed it very much aligns with the intentions of the author.
What about CUE that coexists with non-Go module aware Go code?
The text was updated successfully, but these errors were encountered: