Storing CUE in postgres JSON field

[codewithcheese]

Hi CUE's

At balena.io we want to adopt CUE for an internal document format. Can I please get your thoughts on this rabbit hole?

Internally, we use documents know as contracts, and they contain a mix of JSON and JSONSchema. Plain JSON for concrete values and JSONSchema for abstract values. The contracts are used in many repos, you can think of them as a sort of manifest. The contracts are also stored and queried in PostgreSQL, as jsonb fields.

We would like to use CUE as the contract file representation. JSONSchema is very verbose and large documents become very hard to read and manipulate, CUE is much more concise and would improve the UX when working with contracts.

However, if we use CUE we need a way to store the document in postgres. The most obvious solution would be to use CUE to JSONSchema. However, this appears incomplete at the moment, and we would also need support for importing back to CUE and support for mixed mode data and schema objects.

Since we only need to query on the data, not the schema; we are considering a simplified JSON representation where schema is stored as a CUE string.

Given the following CUE:

title: "example"
envVars: {
  DOMAIN: #Hostname
}

Convert to the following JSON:

{
  "title": "example",
  "envVars": {
    "DOMAIN": {
      "$schema": "example.com/jsonschema-cue",
      "$cue": "#Hostname"
    }
  }
}

I'm planning to implement a custom (en/de)coder.

I am asking as sort of sanity check, does this sound like a reasonable solution for the given problem?

[verdverm]

Sounds like a custom (en/de)coder for a format between JSON and JSONSchema?

How much of the CUE language are you exposing to the contract authors? Imports, disjunctions, and comprehensions? These are tricker parts of the language to handle in Go. Are there any constraints you can put on the problem space?

For the definition conversion, how would your conversion handle #myDef vs "#myDef"

[codewithcheese]

Hey @verdverm good questions. Wanted to spend a bit of time learning how encoders/decodes work before answering.

How much of the CUE language are you exposing to the contract authors? Are there any constraints you can put on the problem space?

Yes, the more comprehensive the better, but the bare minimum would be to encode non-concrete values. That already gives us type and numeric constraints. This seems easy enough to do with cue.Value. Check !value.IsConcrete() and encode using value.Syntax().

However, as far as I understand, using Value would encode the evaluated state. I wish to encode the non-evaluated syntax for later evaluation. So it seems the encoder should be performing something more like an AST rewrite.

But, how to evaluate if an AST value can be encoded as JSON or should be encoded as a CUE string?

Is there a way to map between the AST and the Value's?

[codewithcheese]

Is there a way to map between the AST and the Value's?

I see jsonpb encoded uses LookupPath

[mpvl]

However, as far as I understand, using Value would encode the evaluated state.

Calling Syntax on value, though, will give the original expression, unless explicitly removed with Eval or if options request so (not saying I'm happy with the Syntax API).

So it seems the encoder should be performing something more like an AST rewrite.

[mpvl]

Making a jsonschema generator wouldn't be terribly difficult and mostly entails refactoring the OpenAPI generator to pull out that functionality and deal with the slight differences between the two.