How to set up per-environment configs

[vikstrous]

I'm trying to set up per-environment configs with cue, but I'm running into a bug.

... in paths doesn't work with symlinks.

Repro:

tree
.
├── cue.mod
│   ├── module.cue
│   ├── pkg
│   └── usr
├── environments
│   ├── development
│   │   ├── development.cue
│   │   └── services -> ../../services/
│   └── production
│       ├── production.cue
│       └── services -> ../../services/
└── services
    └── frontend
        └── frontend.cue

10 directories, 4 files

cat environments/development/development.cue
package service

Environment:: "development"

cat environments/production/production.cue
package service

Environment:: "production"

cat services/frontend/frontend.cue
package service

main: "hello \(Environment)"

cue eval ./environments/development/services/frontend/
Environment :: "development"
main:          "hello development"

Expected:

cue eval ./environments/development/services/...
Environment :: "development"
main:          "hello development"

Actual:

cue eval ./environments/development/services/...
cue: "./environments/development/services/..." matched no packages

If there's any info about how to customize configs for different environments or any best practices, please let me know. I've read all of the docs and I didn't see anything mentioning this issue.

[vikstrous]

This fixes the issue:

diff --git cue/load/fs.go cue/load/fs.go
index 082efbb..4af150c 100644
--- cue/load/fs.go
+++ cue/load/fs.go
@@ -243,7 +243,7 @@ var skipDir = errors.Newf(token.NoPos, "skip directory")
 type walkFunc func(path string, info os.FileInfo, err errors.Error) errors.Error

 func (fs *fileSystem) walk(root string, f walkFunc) error {
-       fi, err := fs.lstat(root)
+       fi, err := fs.stat(root)
        if err != nil {
                err = f(root, fi, err)
        } else if !fi.IsDir() {

Is there any reason not to make this change?

[xinau]

@vikstrous I'm not sure if this isn't intended as this could lead to non obvious behavior with larger configurations.
Also you could achive the desired behaviour by using packages.

$ find . -type f      
./cue.mod/module.cue
./services/frontend/frontend.cue
./environments/development/development.cue

$ cat ./cue.mod/module.cue
module: "example.com/pkg"

$ cat ./services/frontend/frontend.cue
package frontend

Environment: string
Main: "hello \(Environment)"

$ cat ./environments/development/development.cue
package development

import "example.com/pkg/services/frontend"

frontend & {
  Environment: "development"
}

Note that the cue.mod directory was generated using cue mod init just the module name in module.cue needed to be changed (this needs to be a domain name I think)

Maybe @mpvl could give a little more insights.

[vikstrous]

I'm not understanding how your example would be extended for non-trivial use cases. Would you have to explicitly list every struct and every package for every environment?

I agree that symlink hacks are a last resort solution, but in this particular case, the symlink structure feels simpler than multiple packages and removes all the repetition.

Could we just restrict symlinks to not be allowed to escape the module and allow them?

Could we come up with some other way to inject an environment-specific config?

This partial solution kind of works:

cue eval ./environments/development/development.cue ./services/frontend/frontend.cue

but you can't specify a file followed by ./services/... because it's a mix of files and directories. Maybe that should be allowed somehow? or there should be some other way to inject a root config?

I was thinking of maybe using _tool.cue files, but that doesn't seem possible.

Another solution is to wrap everything in a shell script that copies the right environment config to the root, but that seems even worse than symlinks.

A related problem: some environments are dynamic (ex. environments created to test pull requests), so how do we inject those values into the config?

I actually come up with another solution that uses the hacky syntax from doc/tutorial/kubernetes/manual/services/k8s.cue. The idea was to render all environment configs at the same time and then use -e to exact only the one you need.

tree .
.
├── cue.mod
│   ├── module.cue
│   ├── pkg
│   └── usr
├── environment_config.cue
└── services
    └── frontend
        └── frontend.cue

cat environment_config.cue
package services

environmentConfig: development: {
  name: "development"
}
environmentConfig: production: {
  name: "production"
}

cat services/frontend/frontend.cue
package services

_frontend: ENV: output: {
  services: frontend: name: "frontend \(ENV.name)"
}

environment: {
  for envkey, envconfig in environmentConfig {
    "\(envkey)": (_frontend & {ENV: envconfig}).ENV.output
  }
}

I'll change the issue title to be about per-environment configs since that's the real issue here. Initially I was hoping that there's a simple solution, but maybe not.

[mpvl]

Symlinks are generally problematic.

Why did the _tool.go approach not work for you? I can imagine it doesn't work because there is currently no easy way to pass values via the command line, but that is fixable.

Allowing a mixture of *.cue files with packages, where the *.cue files get mixed in with each of the selected namesake packages may be an option, but would require some though. Making the _tool.go work seems the most ideal solution to me.

[xinau]

@vikstrous I'm unable to understand why this approach wouldn't scale. To be fair maybe my current configurations written in CUE aren't that large to allow for more valuable insights. I'm curious where my usual approach fails:

1. Often I have an env directory where my environment specific configuration resides (env/production.cue, env/development.cue are different packages).
2. For modular configuration or libraries I've got a lib directory (lib/database.cue, lib/instance.cue, sometimes they are nested).
3. And to "glue" these two together I have a src directory where I import configurations from lib and further specify them, such that I need to define only a small number of values in my environment specific configurations.
   Btw. thanks for the reminder that I could use cue eval src/base.cue env/production.cue instead of importing them.

---

Regarding the following question

but you can't specify a file followed by ./services/... because it's a mix of files and directories. Maybe that should be allowed somehow? or there should be some other way to inject a root config?

I'm not sure if this is you want todo, as it's a little bit more verbose, but you could specify all your services in a services/services.cue file and unify them their or reexport them.

package services

import (
  "example.com/pkg/services/backend"
  "example.com/pkg/services/frontend"
)

Frontend: frontend
Backend: backend
# or
{
  backend,
  frontend
}

I'm looking forward for your input.

[vikstrous]

@mpvl

Tool files have access to the package scope, but none of the fields defined in a tool file influence the output of a package.

I can't have anything in a _tool file affect how the config will be rendered. I'd have to set up something to render all possible configs and then exact the relevant environment's version, which is what I'm already doing in the example I provided in #190 (comment)

@xinau

Based on your suggestions it sounds like the structure you are describing is pretty different from the one in the tutorials? Your structure sounds more like the traditional inheritance based structure, but cue actually doesn't allow you to overwrite anything, so isn't that an issue? The tutorials seemed to advocate for the following structure:

You just have root, intermediate and leaf directories where each one adds more and more specialization. You can put common structures in the root or import them from other packages. Then you unify things as needed at run time in _tool.cue files by passing the cue cmd command a list of packages to act on. That way different subsets of your app can be in different directories that can be deployed independently.

Re: making environments into packages and everything else as a src package

The problem with this approach is that it removes the ability to choose a subset of leaf directories for the tools to act on. You have only one centralized environment specific directory (or cue file as described in the example in #190 (comment)). I don't see how I can keep the ability to specify a subset of packages to act on. I also don't like having to explicitly import everything into a centralized package. That seems to go against the design of cue. I thought that cue was all about not having to explicitly instantiate everything?

Sorry if I'm totally misunderstanding things. I would love to see a more complete example of the best way to structure a cue config tree with different environments.

[mpvl]

@vikstrous / @xinau: I see where @vikstrous is coming from and it is a legitimate use case. This is a classic cross-cutting problem. CUE solves this in the language with aspect-oriented features, but the same problem exists at the file level.

@vikstrous if command line flags could be passed to tool you could select an environment to mix in at the root of the services. Would that solve your use case?

[vikstrous]

Yeah, that would be ideal. It would be very similar in UX to the symlink structure or explicitly selecting a file for the environment.

I think it would be helpful to see a more concrete spec so I can comment on whether or not it selves my use case completely. Reminder: in addition to hardcoded environments (dev, prod), I need to have dynmaically generated environments per pull request, so some type of dynamic non-file input would be necessary as well. Without that, I would still have to wrap cue with a tool to generate files.

[xinau]

@vikstrous and @mpvl sry for the late reply. I'm very grateful for the elaborate input from both of you. As I'm still trying to figure stuff out my self, it seems that even though my solution "works" atm. it might not be sufficient enough for more complex scenarios in the long run.

I think this issue should also result in a documentation page on the cuelang.org site, as this seems to be a more advanced topic, where other people could stumble up on. Maybe it also makes sense to elaborate a little bit more about the aspect-oriented features of cue.

Thank you :).

[mpvl]

An update on the progress here: we are now adding struct and package-level attributes. One possibility would be to have a build tag-like approach. For instance, you would define configuration for your production and staging environments within the same package as anything else, but tag these files respectively with @build(prod) and @prod(stage) attributes. On the command line one would then say:
cue eval ./... -t prod or cue build ./... -t stage for instance. You could have a conflicting definition in the prod and stage files to ensure that the two can never be selected together.

What I like about this approach over having loose packages or files that need to be mixed in, is that the intent to mix in is encoded within the file representation. So a static analyzer could find the @build tags and, by unifying the tagged files only, determine the possible combinations and do a fully scoped analysis across all cross-cutting possibilities. The tooling could also display the possibly combinations of build tags that are available to the user, or even which combination of build tags would produce a concrete instance.

Overall, such a feature fits very well with the aspect oriented nature of configuration.

Note that unlike Go, and to emphasize the aspect-oriented intent of this feature, we would only allow a single build tag in the attribute and would not allow negation. This forces the logic, such as default values, to be worked out in the language, instead of by build tags. Note that the purpose of this would be quite different from Go build tags.

As a nice side effect, this is much easier to type on the command line as well. Question remains whether the command line flag should be --tags/-t to be consistent with Go, or --build/-b to be consistent with the @build attribute.

@rogpeppe @jba @xinau @vikstrous

[mpvl]

One caveat is that in order for this to work in your example, the dev and prod packages would have to be merged at the root directory so that they will be visible throughout the directory hierarchy. This could be accomplished, though, by keeping them as separate packages and then including stub fies at the root directory as such:

// services/prod.cue
@build(prod)
package services

import "example.org/environment/production"

production

// services/dev.cue
@build(dev)
package services

import "example.org/environment/development"

development

Let me know if something like this would work for you.

[vikstrous]

I think the proposal does work for static environments like prod or staging.

1. It's less obvious how it would work for dynamically generated environments, like spinning up a new cluster per pull request. I think the dynamic use case could be handled outside cue by doing a basic recursive string replacement (ex. with sed) as a postprocessing step. Do you have other ideas for how a dynamic environment might work?

2. The import boilerplate stuff makes sense. It makes the wiring explicit and it feels pretty easy to reason about. The effect of choosing one build tag or another is equivalent to copying the imported package into the importing package. It's also very cool how for small projects you can put your environment-specific values directly in services/*.cue files.

edit:
3. I don't have a preference about the build tags CLI argument. --tags is more like go, but it sounds like these tags work sufficiently differently from go build tags (for now?) that using the same name might lead to people making bad assumptions. On the other hand, build --build sounds weird and unhelpful.

[mpvl]

There is one other approach that may help here: specify configuration parts itself on the command line. To improve analyzability, it is probably best if the allowed values are explicitly tagged. For instance,

var: domain: "prod" | "staging" | "test" @tag()

would then allow cue eval -t var.domain=prod. The tag attribute could be of the form @tag(name[,value]), where name is either empty, an alias or -, where the latter treats the possible values as boolean flags (value must be disjunction of strings). value would allow to further constrain the possible values. Values must be something that can be parsed with flags.

Only tags defined in a package itself would be considered: tags defined in imported packages are not.

var: domain: "prod" | "staging" | "test" @tag(env)

could allow cue eval -t env=prod
and

var: domain: "prod" | "staging" | "test" @tag(-) // values must be unique

could allow cue eval -t prod

Advantages both less intrusive and more flexible than build tags. It feels more like CUE. Unlike build tags, it allows for injection of dynamic values on the command line.

The disadvantage is that this is starting to look like the non-hermetic injection of many other config langs. It is not changing the nature of the configuration, however, and having explicit tags, like build tags, makes intent explicit, instead of shoving it under a rug. I expect a similar kind of analysis to be possible as with build tags.

[vikstrous]

This proposal makes sense to me and seems simpler. It wouldn't be necessary to have separate files for different environments any more.

I assume that the first value would be the default?

This still doesn't address the problem of creating per-pull-request environments, but it's simple enough to do a post-processing step in another tool (or shell script) to fix up any constants that include the PR number.

[leoluk]

We have a top-level tool script in a separate package that reads an environment variable using os.Getenv to parameterize the main config.

[mpvl]

@vikstrous

I assume that the first value would be the default?

It was a deliberate choice to not have any defaults in tags, as one can already specify default values in CUE itself. It is a bit more typing, but it means there is only one mechanism to do things. The tag implementation is dead simple in the end.

[mpvl]

@vikstrous @leoluk

This still doesn't address the problem of creating per-pull-request environments, but it's simple enough to do a post-processing step in another tool (or shell script) to fix up any constants that include the PR number.

With the current implementation it is possible to specify arbitrary values. Say

cue export -t foo=bar

But you can indeed also use environment variables.

[mpvl]

@vikstrous @leoluk
There is now also another method one can use: the cue tool now takes - as a file, meaning stdin, stdout or both, depending on the command. This allows for cue fmt -, for instance, but also allows a file with fields to be mixed in. This is not as structures as using tags, though.

Tags don't make all cases nice, so it may still be necessary to have another method. We may still consider @build tags, and we are also looking to support a special import mode that can be overridden on the command line import ":foo" for instance. The latter could be an improvement and generalization for how _tools.cue and _test.cue files would work.

But for now there is probably enough support to get people going. So we can gather some data and see where this goes.

[mpvl]

The latest release now includes the --inject/-t mechanism. This should be general enough to slice and dice configurations.

Two things that may be separately considered are:

• whole-file injection (like Go build tags)
• allowing an -m flag to merge configurations.

I like the first mechanism, as it is in the same spirit as --inject of explicitly specifying in a configuration how it is expected to interact with the command line.

Feel free to open this Issue if field-based injection is not sufficient and whole-file injection is desired.

[cueckoo]

This issue has been migrated to cue-lang/cue#190.

For more details about CUE's migration to a new home, please see cue-lang/cue#1078.