Automation and dashboarding

[mhausenblas]

It would be useful to have current results available in an automated fashion (via GitHub action) and rendered in a nice dashboard. I'm happy to take a stab at it. Please let me know if you think that's useful and if there are any requirements?

[RichiH]

This would depend on #28 at the least, else we end up testing old versions all the time.

I am torn on where such a running log of results should live; maybe https://github.com/prometheus/compliance-results or some such would make more sense?

If it runs in the main repo, we could look at PRs always triggering a test run to commit per-commit-id test results. One concern of this automation would be if a binary can't be downloaded and erreneous 100% FAIL is reported.

Thoughts?

[mhausenblas]

Thanks for the feedback and questions @RichiH!

I don't have a good answer to where the results should stay but you could have a look at what I so far put together in .github/workflows/generate-results.yml which runs once a day dumping it into a sub-dir results/. Note that I've been running this for a few now to let it soak a little and so far no issues.

What I had in mind as the next step was to consume the generated JSON files in a static HTML page with some custom JavaScript to render a nice table. I tried many available tools to turn Go test results into reports but either they didn't quite work or come with huge dependencies, that's why I thought doing some custom rendering would be the best way forward.

[roidelapluie]

It would be great to generate markdown output, therefore we can integrate it in https://prometheus.io