CoCo: Cogent Configuration strategy

[rcoreilly]

This discussion is a place to consider the config file strategy for Cogent Core. If we were to decide we needed some kind of new format, then it could be called CoCo for Cogent Configuration.

We need config files for the core tool to specify packaging etc options for apps, and for the emergent simulations and parameter settings, among various other possible uses. Cogent Home is another possible consideration -- YAML is used extensively in the Home Assistant world, and it is kind of a nightmare. There is some urgency to figure out a plan based on the need for a stable updated emergent v2 framework sooner rather than later.

The currently supported config file format for the cli package is TOML, spec v1.0.0 here: https://toml.io/en/v1.0.0. This was chosen for python here: https://peps.python.org/pep-0518 in part based on this evaluation of options: https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f and it is used extensively in the rust cargo tool: https://doc.rust-lang.org/cargo/reference/manifest.html

The main negative aspect of TOML is the verbosity in dealing with deeply nested structures. There are many tradeoffs here, and this discussion is illuminating: toml-lang/toml#781 It looks like they made some progress here: toml-lang/toml#516 on supporting a more JSON-like syntax for multi-line tables. This was another attempt to deal with the nesting issue: toml-lang/toml#744

The main issue with YAML is its much greater complexity, and StrictYAML is an attempt to deal with that. There is some interesting discussion here: https://hitchdev.com/strictyaml/why-not/turing-complete-code/ -- in particular the many suggestions that, contrary to one of our core principles, code may not always be the best option: there are benefits to making things more strongly constrained.

In terms of organizing the discussion, here some points:

• TOML seems pretty well thought out and, with recent changes, may be the best option. Deep nesting is a challenge for anything, and maybe their solution is actually the best compromise? Also, there is a huge advantage to using a well-documented, widely-supported standard, vs making something new. I would guess at this point that it would be very difficult to overcome that threshold.
• Do we really want to add any actual programmable elements to a config format? Maybe we should just keep those separate, and embed TOML within .goal files that are run in yaegi for dynamic programmable config needs, e.g., in Cogent Home? This capability is not needed for emergent or core tool or most current cli config cases.
• Conversely, the best solution we found for styling is to just write a closure, because then you get completion and strong typing etc. So maybe closures are the best config format, and we just need to find a way to make them more portable as standalone files somehow? The main issue here is creating dynamic structure (slices, maps), and then of course the Make / Plan infrastructure for truly dynamic structures etc.
• Would be good to accumulate more concrete use-cases here to drive further consideration.

[rcoreilly]

The closure-based config file for emergent/axon parameters might look like this:

package config
import "github.com/emer/axon/v2/axon"
func Config(lp *axon.LayerParams) {
    lp.Act.Dt.Vm = 0.015
    ...
}

so it is literally just a minimal .go file (so an editor will have enough basis to give you lookup on the fields) with a function. We just load this fragment into yaegi and append a line that calls the function with the relevant object(s) and it just works :) The package and import are superfluous annoying boilerplate (probably would just cut those b/c yaegi would already be properly config'd with imports).