Continuous Integration

[PeterGreth]

I think your work (#563) can be the basis for something we lack today in Audiveris project: a "continuous integration" à la Jenkins.
-- Hervé

I would not say that I'm an expert in CI, but I'm certainly interested in gaining experience.

A first starting point would be automatic builds and automated execution of unit tests.

I think this could easily be achieved using Github Actions.

Do you have specific ideas? Any input is welcome 😃

[hbitteur]

Forget about the existing unit tests, they are way too basic.
Processing the data/examples files is I think much more interesting and useful.

I have no special idea about CI except that we should keep it as simple as possible. Running your result evaluation on the dozen of examples scores is for me the good objective to start with. We'll see later if we could/should extend it, but first let's make it work easily.

1/ We should be able to run it manually, at least before a release is published (hence the need for something easy to launch).

2/ Also, could we run it, automatically this time, when a new commit is pushed to github? On "master" branch, on "development" branch? Not on the other branches, which are meant for temporary developments, see our wiki article.

This would be a good start.

[Altonss]

Github actions would be a very good start: build/tests could be run on each commit and even on PRs to check if it doesn't break something ;)

[hbitteur]

@PeterGreth
Have you made progresses on this topic?

I have a suggestion:
I have been working with DeepScoresV2 dataset, in order to train a patch classifier on many samples.
As a matter of serendipity :-), I discovered that the dataset also provides ground truth information for heads and rests. They call it "beat information", which roughly means on which measure beat a given head/rest is located.
Although the dataset contains just synthetic images and this beat information covers "only" rhythm, it is probably the most valuable piece in OMR results.
So, we could consider including a bunch of these "annotated sample scores" in the integration tests...

[PeterGreth]

Hi Hervé

Any progress on CI?

I played around a bit with CI and that seems to work. See also master...PeterGreth:audiveris:master. The main problem is that there is no progress in the integration tests (#563), so CI currently would only execute unit tests. I probably have time on the weekend to push this a little bit.

If I remember correctly there were still some questions left unanswered in #563. Maybe by chance you have time to answer them? :)

Suggestion: DeepScoresV2

Thanks for the suggestion, sounds promising! I will have a look.

[NohWayJose]

I don't know if this (https://galasa.dev/about/automation) might be helpful. It's an Open Source test automation framework that I worked on (as a UX Designer) a couple of years ago. Worth a quick read. It means you can concentrate on development and still deliver a robust and extensively tested product. It's scalable right up to big enterprise products and helps with Ci/CD DevOps.

[hbitteur]

@NohWayJose
I think the problem is not with the test framework, but rather with the tests themselves.
We have only basic unit tests and no integration tests.

It's difficult to imagine a working integration test (apart from checking if the app crashes or not, of course).
Say we have modified something and the processing of a given test score now gives slightly different results.
Are they better or worse?