Fix flaky tests

[hofi1]

com.github.jk1.license.filter.LicenseBundleNormalizerSpec."The regular expression must be matched"

Problem

The test com.github.jk1.license.filter.LicenseBundleNormalizerSpec."The regular expression must be matched" asserts two JSON strings for equality. The corresponding data is stored in sets. But the order, in which the data is returned by the Sets, is not deterministic. The test expects the values to be in certain order, even tho two JSON String are considered as equal if the data is in a different order (regarding the specification of JSON).
This leads to a flaky test.

This problem was found by the NonDex Engine.

Gradle-License-Report/src/main/groovy/com/github/jk1/license/Model.groovy

Lines 39 to 48 in 5e273f9

	@Sortable(includes = ["group", "name", "version"])
	@Canonical
	class ModuleData {
	String group, name, version
	Set<ManifestData> manifests = new TreeSet<ManifestData>()
	Set<LicenseFileData> licenseFiles = new HashSet<LicenseFileData>()
	Set<PomData> poms = new HashSet<PomData>()
	boolean isEmpty() { manifests.isEmpty() && poms.isEmpty() && licenseFiles.isEmpty() }
	}

Solution

The best solution would be to use JsonAsserts. But since it is unlikely that the owern of the repo would accept a new dependency, it was fixed by making sure the data is returned in a deterministic order.
By changing the HashSets to TreeSets (to guarantee the order of the elements in the JSON string) in the ModulData class and adding the @Sortable Annotation (needed by the TreeSet to order the elements in the set) to the PomData class, determinism is established.

Gradle-License-Report/src/main/groovy/com/github/jk1/license/Model.groovy

Lines 39 to 48 in 140ee49

	@Sortable(includes = ["group", "name", "version"])
	@Canonical
	class ModuleData {
	String group, name, version
	Set<ManifestData> manifests = new TreeSet<ManifestData>()
	Set<LicenseFileData> licenseFiles = new TreeSet<LicenseFileData>()
	Set<PomData> poms = new TreeSet<PomData>()
	boolean isEmpty() { manifests.isEmpty() && poms.isEmpty() && licenseFiles.isEmpty() }
	}

Gradle-License-Report/src/main/groovy/com/github/jk1/license/Model.groovy

Lines 57 to 64 in 140ee49

	@Canonical
	@Sortable(includes = ["name", "description", "projectUrl", "inceptionYear"])
	class PomData {
	String name, description, projectUrl, inceptionYear
	Set<License> licenses = new TreeSet<License>()
	PomOrganization organization
	Set<PomDeveloper> developers = new TreeSet<PomDeveloper>()
	}

com.github.jk1.license.ProjectBuilderSpec."it creates modules with license-files"

Problem

The test com.github.jk1.license.ProjectBuilderSpec."it creates modules with license-files asserts that two lists contain the same elements. These elements are stored in a List. But the order, in which the elements in the data structure are returned by the set, is not deterministic. The test expects the values to be in certain order.
This leads to a flaky test.

This problem was found by the NonDex Engine.

Gradle-License-Report/src/test/groovy/com/github/jk1/license/ProjectBuilderSpec.groovy

Lines 119 to 121 in 5e273f9

	data.configurations*.dependencies.flatten().poms.flatten().find { it.name == "pom1" }.licenses as List == [APACHE2_LICENSE()]
	data.configurations*.dependencies.flatten().poms.flatten().find { it.name == "pom2" }.licenses as List == [MIT_LICENSE()]
	data.configurations*.dependencies.flatten().poms.flatten().find { it.name == "pom3" }.licenses as List == [LGPL_LICENSE(), MIT_LICENSE()]

Solution

By sorting the elements in the lists, a deterministic behavior can be guaranteed.

Gradle-License-Report/src/test/groovy/com/github/jk1/license/ProjectBuilderSpec.groovy

Lines 211 to 213 in 140ee49

	data.configurations*.dependencies.flatten().licenseFiles.flatten().fileDetails.flatten()*.file.sort() ==["file1", "file2", "file3"].sort()
	data.configurations*.dependencies.flatten().licenseFiles.flatten().fileDetails.flatten()*.license.sort() == ["lic1", "lic2", "lic3"].sort()
	data.configurations*.dependencies.flatten().licenseFiles.flatten().fileDetails.flatten()*.licenseUrl.sort() == ["licUrl1", "licUrl2", "licUrl3"].sort()

Reproduce

To reproduce follow the steps:

1. Run ./gradlew build -x test
2. Add the following text to the top of the build.gradle file in $PROJ_DIR.

buildscript {
    repositories {
      maven {
        url = uri('https://plugins.gradle.org/m2/')
      }
    }
    dependencies {
      classpath('edu.illinois:plugin:2.1.1')
    }
}

3. Add the following line to the end of the build.gradle file in $PROJ_DIR.

apply plugin: 'edu.illinois.nondex'

4. Run

./gradlew --info nondexTest --tests=_full.qulified.identification.of.test_ --nondexRuns=50

[jk1]

@hofi1 thank you for your contribution. I suppose that the tests are not actually flaky. The reason being, a HashMap ordering is not guaranteed, but is still stable from run to run. Yet it's definitely not something to rely upon in the long run.