Add nightly check for Product and Community registries

[thepetk]

What does this PR do?

This PR creates a new workflow check_registry.yaml which will be scheduled for every day at 03.00 UTC. In this workflow we are trying to run an E2E test by running alizer against a stack's starterprojects. We expect that the devfile of this stack will be inside the list of alizer's matchings.

The registries we are checking are:

func getRegistries() []DevfileRegistry {
	return []DevfileRegistry{
		{
			Name:   "Community Registry",
			Url:    "https://registry.devfile.io",
			Filter: model.DevfileFilter{},
		},
		{
			Name:   "Product Registry",
			Url:    "https://devfile-registry.redhat.com",
			Filter: model.DevfileFilter{},
		},
	}
}

In order to create a complete JSON for all registry entries we are running the check_registry.go file which is responsible to gather all details from the given registries:

• First it GETs the <registryUrl>/v2index response and for each stack/sample fetched tries to getch the devfile details:

starterProjects, err := getStarterProjects(fmt.Sprintf("%s/devfiles/%s", registry.Url, devfileType.Name))

• If a stack has different layout and we don't have available the git.remote.origin we are using the getProjectReplacements method.

• After the JSON is generated the script checks every entry, clones the given repo, runs alizer against it and checks the results.

• If the expected devfile is not inside the list then the process exit 1.

Which issue(s) does this PR fix

fixes devfile/api#1165

PR acceptance criteria

Testing and documentation do not need to be complete in order for this PR to be approved. We just need to ensure tracking issues are opened.

• Unit/Functional tests

• Documentation

How to test changes / Special notes to the reviewer

Locally you can try to run

bash test/check_registry/check_registry.sh

There is also a test run here: https://github.com/thepetk/devfile-alizer/actions/runs/5681161565

[codecov]

Codecov Report

Patch coverage: 45.77% and project coverage change: +28.64% 🎉

Comparison is base (7f521b2) 37.91% compared to head (9f73689) 66.56%.

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #15       +/-   ##
===========================================
+ Coverage   37.91%   66.56%   +28.64%     
===========================================
  Files           9       10        +1     
  Lines        1221     1358      +137     
===========================================
+ Hits          463      904      +441     
+ Misses        729      394      -335     
- Partials       29       60       +31     

Files Changed	Coverage Δ
pkg/apis/recognizer/devfile_recognizer.go	53.62% <20.00%> (+53.62%)	⬆️
test/check_registry/check_registry.go	46.71% <46.71%> (ø)

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mike-hoang]

Nice! Overall looks good, just a few nits that I think should be addressed

[thepetk]

Nice! Overall looks good, just a few nits that I think should be addressed

All items have been addressed @mike-hoang :)

[thepetk]

@mike-hoang I've gone through again the workflow (I had in mind what you said about the cleaning process) and I made some further fixes:

1. Clean dir in case of failure
2. In case of failure print the alizer output and command used
3. Fix the loop for for raw_selected_devfile_name in $(echo $alizer_output | jq -c '.[].Name') in order to skip check if found_matching is already 1.
4. Excluded the java-maven (too generic) and the websphere stacks (they use the same projects as open-liberty.

For the websphere stacks I've created this item devfile/api#1206

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mike-hoang, thepetk

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment