CI specific configuration file validation

[lorenzwalthert]

Is there any way to validate if a travis / appveyor script is still complying to a template? My hypothesis of how things could fail:

• travis changes something in their default specification, so to continue to use the current specification, we now need to explicitly specify the key value (e.g. an environment variable). However, all projects using tic have a .travis.yml file in their root that has no link to an upstream repo from which it could get updated. Hence, there is no way for these projects to update their CI, in worst case, they will fail silently or loudly.
• appveor depreciates a stage / keyword in their markup language. When hard depreciation sets in after soft depreciation, all projects that use tic will fail with an invalid YML specification error.

To prevent this from happening, tic should check if the ci file such as .travis.yml are up to date with a template that is stored somewhere (ideally this check could be part ofa simple pre-push hook). If it is not, we should ask people to kindly update their yml file for the respective service. Also, since tic.R should contain all abstractions, there would actually be no reason for anyone to have specific information in the .travis.ci, correct? Maybe my solution is an overkill though. What do you think?

[pat-s]

Is there any way to validate if a travis / appveyor script is still complying to a template? My hypothesis of how things could fail:

Somewhat - we have the example repos running with the template. We would recognize if they start failing.

travis changes something in their default specification, so to continue to use the current specification, we now need to explicitly specify the key value (e.g. an environment variable). However, all projects using tic have a .travis.yml file in their root that has no link to an upstream repo from which it could get updated. Hence, there is no way for these projects to update their CI, in worst case, they will fail silently or loudly

This is true. Even though this is not very likely, it is ofc possible.
Also we might make some opinionated changes to the template and should have the option to distribute those easily. Something along the lines of your pre-commit pkg but R based.

Auto-updating is problematic since it may break existing workflows. And unfortunately, Travis does not allow referring to an external source for .travis.yml yet. This would simplify a lot.

To prevent this from happening, tic should check if the ci file such as .travis.yml are up to date with a template that is stored somewhere (ideally this check could be part ofa simple pre-push hook).

As said, this is complicated. If the user made local changes, e.g. to include additional system libs, we would overwrite those.

Also, since tic.R should contain all abstractions, there would actually be no reason for anyone to have specific information in the .travis.ci, correct?

Not 100% sadly. System lib installation is always the problem. This differs for each OS system and cannot be done within R. Hence the user needs to interact with the respective CI yaml file. Even if we would forward this externally, the names of such libraries would differ e.g. on Ubuntu/Windows/macOS.

[pat-s]

I thought about this for a while. A function providing a manual update approach using the template files stored on GH would definitely be useful.
This would work for 90% I guess.

For all others - I am wondering if there is a way that this function could act smart in that it would scrape potential system libs added by the user and readds it to the YAML after updating.
This would make updating a smooth experience.

[pat-s]

Regarding CI YAML validation:

CI systems usually have their own linter which can be accessed via the API, e.g. travis lint (via the travis cli gem).
We could think of incorporating this in some way. However, I think the approach in #194 should be more user-friendly.

To prevent this from happening, tic should check if the ci file such as .travis.yml are up to date with a template that is stored somewhere (ideally this check could be part ofa simple pre-push hook).

See #194 for a sketch regarding updating local YAML files.
A pre-commit hook would be possible once we have a robust implementation of this.
In this case more thought would be needed on how to prevent applying the update every time (which will cause a file mod and then also a fail of the pre-commit hook).

[pat-s]

I'd conclude that evaluating YAML files against upstream changes of the CI provider is out of scope for {tic}. Some providers do not even provide a yaml validation mechanism via their API.

I think most of this part can be solved by implementing an update mechanisms as proposed in #194. Also we should recognize early enough in {tic} and users should maybe update once in a while.