Modules

[rogpeppe]

This is an umbrella discussion where people can post their thoughts on how they'd hope or expect to use CUE modules. In particular, concrete use cases would be much appreciated.

[dontlaugh]

We have a cli utility, written in Go, that uses the Go standard library's embed package to embed a large (300+) tree of CUE files. An entire cue.mod, essentially. The tree of files is varied: our libraries plus some libraries authored by others.

We will always embed a CUE module into the binary, because it's a nice delivery mechanism for air-gapped environments. But for everyone else we want to have a native versioning concept (remember, we have multiple library authors down there), and the ability to fetch updates over the network.

We talked about this on a call a while ago, but I thought I'd share it here. ❤️

[verdverm]

Do you mean you want your tool to be module aware, and have the same dependency management exposed as CUE does, for consistency? Which means your users have one less binary to worry about, they only need your tool, not cue to fetch dependencies?

Or another way to say this, the cue Go API should expose the dependency management workflows so that tools building on CUE can make these workflows available through their own CLIs?

This is something I would like

[verdverm]

On today's community call (#2219), several features for the module system were outlined.

Is there an updated proposal that can be commented on? It seems like there is already implementation before the proposal, which is the opposite order for other major initiatives and changes.

[verdverm]

Using the OCI format for modules & packaging, publishing and consuming, is very interesting

It would allow for using a standardized format and protocol tools are starting to adopt. It would mean that CUE does not have to create a format, protocol, or the code & infra for supporting a module ecosystem. Many existing tools in a broader ecosystem could be leveraged for day-2 issues as well.

I could imagine 3 layers for module.cue, module content, and provenance (like helm or a sum file). Having the module.cue in a separate layer would facilitate running MVS to resolve versions without needing to download the full module content.

• https://github.com/opencontainers/distribution-spec/blob/main/spec.md
• https://helm.sh/docs/topics/registries/ | https://helm.sh/docs/topics/registries/#helm-chart-manifest
• https://goharbor.io/docs/2.7.0/administration/user-defined-oci-artifact/
• https://medium.com/@dustalov/how-homebrew-serves-52m-packages-per-month-413b9f0cf685

[dontlaugh]

I like this as an option. Flux optionally does this, too https://fluxcd.io/flux/cheatsheets/oci-artifacts/

So perhaps we think of it as optional transport. OCI Repositories are a pain to run, however. I wouldn't want to end up relying on OCI internals as an absolute requirement.

[verdverm]

We use GCP, AWS, Azure registries so we don't have to run them :] They already have public & private support with authentication that works with all the things.

Every package host has issues, NPM & PyPI are notable in our build failures. I have a hard time believing that building a new hub from scratch would be less effort or more reliable than using the tools we already have.

My personal preference is to just use git like Go already does. This has been the most reliable module system we use and already supports the pub/priv/auth features we need without needing to reinvent as well. I again don't see how exec'n out to git is harder than making a new protocol, format, and the infra to go with it. Much of the effort has already been done by the Go team, and given CUE is using Go as a model, it would seem natural to leverage code that already exists and has extensive testing.

[DavidGamba]

OCI repositores have the advantage of features like immutable tags out of the box. Go ended up having to build a proxy server to avoid issues with tags getting mutated and repos from dissapearing.
Vendors already support scanning OCI repositories for vulnerabilties as well.
Creating private or public ECR repositories for helm templates takes very little effort with Terraform for example.

This is a doc as to how easy it is on AWS:
https://docs.aws.amazon.com/AmazonECR/latest/userguide/push-oci-artifact.html

There are OCI repositories in virtually any cloud provider too.

[verdverm]

Harbor.io may support custom artifacts too: https://goharbor.io/docs/2.7.0/administration/user-defined-oci-artifact/

[mxey]

I saw “Will use "hub" server rather than VCS directly (most common approach).” mentioned on the slides and I want to say, I really would prefer the VCS approach that Go uses because it makes everything simpler and easier and does not need additional infrastructure.

[mxey]

As for a use case example: We have some reusable CUE code called "kubekit" for our Kubernetes objects. Right now we use Git submodules for “importing” it. The biggest downside is that we have to specify a commit as opposed to a tag. Combined with Renovate auto-updating the downstream repositories, we have a lot of churn in those, because every commit in kubekit creates a commit in the downstream repositories.

[b4nst]

We currently use a simplified version of the go module algorithm. This does look like:

// cue.mod/module.cue
module: "github.com/loft-orbital/cue-module"

require: [
    {
        path:    "github.com/loft-orbital/cue-lib"
        version: "v0.10.0"
    },
]

godef: [
  // Kubernetes
  "k8s.io/api/apps/v1@v0.23.4",
  "k8s.io/api/autoscaling/v2beta2@v0.23.4",
  "k8s.io/api/batch/v1@v0.23.4",
  "k8s.io/api/core/v1@v0.23.4",
  "k8s.io/api/networking/v1@v0.23.4",
  "k8s.io/api/rbac/v1@v0.23.4",
  "k8s.io/api/policy/v1@v0.23.4",
]

Then we leverage cuebe mod commands to vendor and generate dependencies. It's still brittle, but gives us enough material to work.

[kajogo777]

How we implemented package management for CUE, it's not perfect, it's distributed (git-based), doesn't support locking yet (but you can pin commits), and doesn't do nested dependency resolution.
https://docx.guku.io/docs/package-management

I'm wondering about the tradeoffs of using git-based vs registry/hub based package distribution.

we used git because it required no additional effort to publish packages, and we can reuse git versioning.

[verdverm]

It would be nice if dependency modules with tools have a way to make them runnable from the importing project. This is similar to installing a node development tool from npm packages per project.

cue run dependencyCommand

This could be constrained by having a top level bin directory, or in cue.mod/bin, with _tool files that are runnable. There could be conflicts, what does npm do when this happens?

[myitcv]

This proposal was superseded by #2939. As we updated in #2939 (comment), that proposal (and its sub-proposals) have now been accepted, hence we are closing this discussion to represent the decision on the v3 proposal.