testing framework

[roidelapluie]

Hello,

In my research about Scuemata, I have not seen a test framework being mentioned.

It would be great to be able to provide some data, and validate that schema migrations and lacunaes are generates accordingly. Instead of everyone implementing it, an opiniated framework built into scuemata would be useful.

[sdboyer]

An excellent idea, thank you for the suggestion! I have been thinking about this, and am actually a bit hung up on it, and would welcome discussion.

My main issue is that i'm not sure where the boundary should properly be between:

1. Testing scuemata itself (like, the tests in this repo, run on CI).
2. Natively expressed constraints that enforce intended invariants. This almost certainly depends on Proposal: core builtin extensions cue-lang/cue#943. (trivial example)
3. Tools for scuemata authors to use that verify their scuemata are well-formed.

My thought is that as much as possible should be done in category 2, as having constraints expressed there will simply make misuse impossible.

That makes category 1 - "testing" in the scuemata repo itself - kinda weird. Like, what do we test? The simple answer seems to be "write some scuemata which should be valid, and some which should be invalid, and check that that expectation is met." Which is why i've started creating these exemplars, which i intend to also use as part of the docs i'm working on. I've yet to figure out how to fit those into unity, but that is the plan.

But in general, it seems like anything checked in category 2 can be ignored in category 3, and we should prefer that outcome wherever possible. Where it's not possible, though, i think a test framework would be welcome. We haven't gotten to it with our use of scuemata in Grafana itself yet, because the preliminary approach there still doesn't fully check category 2.

Thus far, every time i've thought of some checkable property that seems like it might be category 3, i've tried and succeeded at making it generic/category 2, and been happy with the result. For example, that's how JoinSchema came to be - it was originally expressed ad hoc, with no well-defined relationship to the structure of the lineage itself.

I'd love thoughts about where this line ought to be, especially if they're accompanied with imagined use cases for scuemata :)

[roidelapluie]

I think the assumption is 3 is that you think about it in a "syntax" approach.

This is more, validating that a scuemeta can correctly bring a defined set of data from version X to version Y. I am thinking about a framework for scuemata schema authors. Even if the syntax is correct and enforced, it does not mean that the cue code is correct and is behaving as expected.

[sdboyer]

validating that a scuemeta can correctly bring a defined set of data from version X to version Y.

Ah yes, this is absolutely foundational. The main issue here is the word "correct" - when it comes to mapping one schema to another, there's little we can say generically about what constitutes a correct mapping. For example, if you had (from this exemplar):

Seqs: [
    {
        schemas: [
            {
                before: string
                unchanged: string
            }
        ]
    },
    {
        schemas: [
            {
                after: string
                unchanged: string
            }
        ]
    }
}

Would it be a bug, to be caught via some test in the testing framework you suggest, if the lens did not map before to after? Or if unchanged was not mapped to itself in the new schema? Does the answer change if a lacuna is emitted involving any of those fields?

I have some nascent thoughts about the fundamental maths at issue here, but thus far it's seemed most productive to me that we play out these questions in the context of some real examples. So, if you have some examples, even early and handwavy ones, please offer them, and we can figure out what approach makes the most sense to handle them!

[sdboyer]

i think the new docs on mapping CUE to Go should help explain how i see verification by authors fitting in. Still hacking at some of the code that's referenced in that doc, but so depending on how soon you try it out, it may flake out :)

If you think what's described in that doc adequately obviates the need for a testing framework as you'd thought of it, let's close the issue. If not, i'm curious to hear what's missing!