Skip to content
This repository has been archived by the owner on Sep 26, 2023. It is now read-only.

cmd/unity: support for version-based conditions #57

Open
myitcv opened this issue Mar 12, 2021 · 0 comments
Open

cmd/unity: support for version-based conditions #57

myitcv opened this issue Mar 12, 2021 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@myitcv
Copy link
Contributor

myitcv commented Mar 12, 2021

#55 demonstrates a case where a fix in CUE revealed a bug in the expectations for the Vector project. Up until that PR therefore, tip of CUE would fail against unity.

It's clearly not sustainable to have a broken unity build where we need to remember "oh that's fine because project XYZ's expectation is wrong, we fixed something in CUE".

Neither is it workable to expect project XYZ to move immediately to the version of CUE that correspond to the fix.

Nor is it wise to simply skip such projects from the corpus tests of later CUE versions until project XYZ updates to a later CUE version: we simply lose coverage this way.

Therefore, we need to support bridging multiple versions, and support expectations corresponding to the fixes that have been introduced since the base version of CUE used by a project.

We probably end up with a workflow something like this:

  • fix identified and proposed in CUE CL
  • untiy build run as part of CL trybot fails
  • a temporary overlay update for project XYZ is prepared and tested (locally?)
  • the overlay update is merged in unity
  • the CL trybots can now pass
  • the change can be submitted
  • the temporary overlay change is submitted as a PR to project XYZ
  • the temporary overlay is removed

The challenge here comes in supporting multiple expectations. Because the base version of CUE against which project XYZ is tested still needs to work. Hence our expectations in testscript-style tests need to be made conditional.

Question: how do these conditions work in the context of Go tests and #41? We have far less control over tests in that situation, indeed there is really no concept of an overlay at all. Answer is likely they don't work and we do have to skip Go tests in such situations, with some sort of detail in the tests manifest on the CUE version (ranges) for which Go tests are not run.

For the testscript-style tests we can use versioned-based conditions to gate whether commands are run or not. For example:

# Eval
cue eval ./...
[! v0.3.0-beta.6] cmp stdout stdout.pre_v0.3.0-beta.6
[v0.3.0-beta.6] cmp stdout stdout.v0.3.0-beta.6

Based on the version of CUE under test X, such a version-based condition would be satisfied (i.e. the command run) if the commit corresponding to X is a descendant of the condition version, in this case v0.3.0-beta.6.

The versions supported in these conditions would be a subset of the versions understood by unity:

Implementation notes:

  • the ancestry determination should be done via the git merge-base --is-ancestor <maybe-ancestor-commit> <descendant-commit> command
  • the check should be run within the user cache dir copy of the cue repo
  • a lockedfile lock (same lock file as used elsewhere in unity) should be held whilst checking and conditions
  • because actual tests which evaluate conditions are run within Docker in safe mode, we will need to pre-compute the condition answers and pass the results as a JSON map to the child process
  • this will therefore require that we separately parse and walk each condition to know which version conditions will be tested. Should be relatively straightforward to copy the testscript code that does this
  • we will also need to extend the version resolver interface to return the commit for a given version. If we cannot definitively resolve a clean commit for a given version (e.g. PATH) then we should error in case evaluation against a version condition is encountered
@myitcv myitcv added the enhancement New feature or request label Mar 12, 2021
@myitcv myitcv added this to the Next milestone Mar 12, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Development

No branches or pull requests

1 participant