tests: run test against actual schema files, not snapshots

[jkowalleck]

Current implementation of java tests run against a snapshot of the schema files, shipped with org.cyclinedx java package.
This is undesired and error-prone, as changes to the schema files in this repo are not checked against the schema example files.
Therefore, most java checks in pull requests are just bogus results without any meaning.
This must change ASAP.

Use cases:

• a PullRequest is opened, that modifies a schema file. tests must run based on these changes, not some snapshots.
• a PullRequest is opened, that modify some test-resources. tests must run based on these changes.

acceptance criteria

• all test run against the actual schema files in this repo, not against any other snapshots
• optional, but preferred: tests run offline, do not load any data from external sources
• all currently running tests still pass

scope

• test resources: https://github.com/CycloneDX/specification/tree/master/tools/src/test/resources
  • tests for JSON files
  • tests for XML files
  • text for ProtoBuf files
  • valid-*.{json,xml,textproto} should pass schema validation
  • invalid-*.{json,xml,textproto} should fail schema validation
• schema files: https://github.com/CycloneDX/specification/tree/master/schema
  • XSD for XML
  • JSON schema for JSON
  • proto for textproto

targets

• Java tests -- open, uses remote snapshot which is about to be changed
  • see

    specification/tools/src/test/java/org/cyclonedx/schema/JsonSchemaVerificationTest.java

    Lines 60 to 61 in d37a107

    	final JsonParser parser = new JsonParser();
    	return parser.isValid(file, version);

  • see

    specification/tools/src/test/java/org/cyclonedx/schema/XmlSchemaVerificationTest.java

    Lines 64 to 65 in d37a107

    	final XmlParser parser = new XmlParser();
    	return parser.isValid(file, version);

out of scope (since already fixed/done)

• PHP tests -- done, uses local versions
  • see

    specification/tools/src/test/php/json-schema-functional-tests.php

    Lines 15 to 16 in d37a107

    	define('SCHEMA_DIR', realpath(__DIR__ . '/../../../../schema'));
    	define('SCHEMA_FILE', SCHEMA_DIR . '/bom-' . TESTSCHEMA_VERSION . '.schema.json');

  • see

    specification/tools/src/test/php/json-schema-lint-tests.php

    Lines 5 to 6 in d37a107

    	const SchemaFileGlob = '*.schema.json';
    	const SchemaDir = __DIR__.'/../../../../schema/';

    
    

    specification/tools/src/test/php/json-schema-lint-tests.php

    Line 16 in d37a107

    $schemasFiles = glob(SchemaDir.SchemaFileGlob, GLOB_ERR);

  • see

    specification/tools/src/test/php/xml-schema-functional-tests.php

    Lines 15 to 16 in d37a107

    	define('SCHEMA_DIR', realpath(__DIR__ . '/../../../../schema'));
    	define('SCHEMA_FILE', SCHEMA_DIR . '/bom-' . TESTSCHEMA_VERSION . '.xsd');

• JS tests -- done, uses local versions
  • see

    specification/tools/src/test/js/json-schema-functional-tests.js

    Lines 24 to 25 in 2003555

    	const schemaDir = join(_thisDir, '..', '..', '..', '..', 'schema')
    	const schemaFile = join(schemaDir, `bom-${testschemaVersion}.schema.json`)

  • see

    specification/tools/src/test/js/json-schema-lint-tests.js

    Line 16 in 2003555

    const schemaDir = join(dirname(fileURLToPath(import.meta.url)), '..', '..', '..', '..', 'schema')

• ProtoBuf tests -- done, uses local versions
  • see

    specification/tools/src/test/proto/test.sh

    Lines 104 to 130 in 19a1530

    	function validate() {
    	FILE="$1"
    	SCHEMA_VERS="$2"
    	SCHEMA_FILE="bom-${SCHEMA_VERS}.proto"
    	MESSAGE="cyclonedx.v${SCHEMA_VERS/./_}.Bom"
    	echo ">> validate $(realpath --relative-to="$PWD" "$FILE") as ${MESSAGE} of ${SCHEMA_FILE}" >&2
    	# this test method is a bare minimum, and it might not detect all kinds of malformed input.
    	# could be improved by utilizing protoc -- see https://github.com/CycloneDX/specification/pull/385/commits/8db0967c11cb913ac3c7a9a037159338df3f3bd9
    	docker run --rm \
    	--volume "${ROOT_PATH}/${SCHEMA_DIR}:/workspace/${SCHEMA_DIR}:ro" \
    	--volume "${FILE}:/workspace/test_res:ro" \
    	--workdir '/workspace' \
    	bufbuild/buf:"$BUF_IMAGE_VERSION" \
    	convert "${SCHEMA_DIR}/${SCHEMA_FILE}" \
    	--type "$MESSAGE" \
    	--from 'test_res#format=txtpb' \
    	--to /dev/null
    	}
    	shopt -s globstar
    	for test_res in "$ROOT_PATH"/"$TEST_RES_DIR"/*/valid-*.textproto
    	do
    	SCHEMA_VERS="$(basename "$(dirname "$test_res")")"
    	validate "$test_res" "$SCHEMA_VERS"
    	done

---

caused by #255
related: https://cyclonedx.slack.com/archives/G01PMU9ERJA/p1687363757137109

[jkowalleck]

@CycloneDX/core-team FYI

@CycloneDX/java-maintainers can you help out and fix the tests?

[stevespringett]

The latest schema from the 1.5-dev branch has been incorporated into the latest Core Java 7.4.0-SNAPSHOT

[jkowalleck]

re: #256 (comment)
this does not solve the following purpose: a PR modifies the schema files. tests should run on these changes, not some snapshot. here is such a case: #255

[jkowalleck]

@mr-zepol mentioned he might be able to help out here.

[jkowalleck]

the situation with java tests is so annoying.
i am going to tackle this, now.

[ppkarwasz]

the situation with java tests is so annoying. i am going to tackle this, now.

I can look at it in the weekend.

[jkowalleck]

the situation with java tests is so annoying. i am going to tackle this, now.

I can look at it in the weekend.

this would be fantastic.
i started butchering some code together and realized, that I did not do any java in the past 20 years. so i am not very efficient.

I'll assign the ticket to you, @ppkarwasz , in hope you'll find time for it :D

[jkowalleck]

Hi @ppkarwasz ,
Just wanted to check back on this. Did you find some time for this ticket?
No pressure, I know how busy we all are.

If you have no time for it, just let me know, and I will set the ticket back to "need help", just let me know :)

[ppkarwasz]

I am recovering from FOSDEM, but I put this item at the top of my TODO list for this weekend.

[ppkarwasz]

@jkowalleck,

I fixed the tests in #592.