Schema Generation Test Suite Schematron and Unit Tests Fail, XSLT Relative Import Paths Incorrect

[aj-stein-nist]

Describe the bug

With the current iteration of the unit tests at usnistgov/metaschema@c7b3733, with test-suite/schema-generation/run-tests.sh, the Schematron tests and actual unit tests will fail. The compiled Schematron and unit test XSLT transforms expect the xsl:import directives to frequently look up a directory with <xsl:import href="../imported_transform.xsl"/>. These imports are problematic because the compiled Schematron and some helper files imported refer to the original path structure outside of the scratch directory. This causes immediate failures in all the testing.

Who is the bug affecting?

NIST OSCAL developers ensuring core Metaschema constructs are current and valid, prior to their use downstream in OSCAL.

What is affected by this bug?

Properly testing schema generation, profile resolution, and converter generation as implemented by the XSLT-M4 toolchain.

When does this occur?

Consistently.

How do we replicate the issue?

1. Clone the directly.
2. Initialize all submodules.
3. Switch to the relevant subdirectory. (pushd test-suite/schema-generation)
4. Execute the test harness. (bash -x ./run-tests.sh)

Observe a series of failures such as these:

Expected behavior (i.e. solution)

Paths and imports can be handled contextually (with <xsl:import href="../" in Oxygen and local dev but overridden with specific scratch dir paths as needed for others).

[aj-stein-nist]

So the schema generation test suite script doesn't work, to be specific. We could remove that as a short-term bandaid, but a more substantive fix, but we have a few more options.

One of the better recommendations is to run the XSLT-based transforms with Saxon (the way it is currently designed to do).

[aj-stein-nist]

I will work on this with Dave in Sprint 61. Moving to Sprint 61.

[wendellpiez]

Is this an XSLT M4 issue? Appears to be CI/CD.

[aj-stein-nist]

Is this an XSLT M4 issue? Appears to be CI/CD.

It happens in CI/CD (GitHub Actions). Either way, it is broken wherever it is run. It pertains to XSLT-M4 implementation. That is why I put the XSLT label in October. Let me know if you have any questions, comments, or concerns.

[wendellpiez]

It appears to be a dependency left behind when a file is copied? In this case it appears to be an XSLT that is required by a Schematron at runtime, which is expected to be found but isn't in a /tmp directory.

I'm not sure what the best solution is here. Indeed, the Metaschema Schematron (which examines composition results) must run the entire Metaschema pipeline in order to have results to look at. The problem here is arguably not on the Schematron side, but either in the CI/CD or in achieving composition more easily in the context of Schematron so as to avoid such a dependency.

[aj-stein-nist]

It appears to be a dependency left behind when a file is copied? In this case it appears to be an XSLT that is required by a Schematron at runtime, which is expected to be found but isn't in a /tmp directory.

I'm not sure what the best solution is here. Indeed, the Metaschema Schematron (which examines composition results) must run the entire Metaschema pipeline in order to have results to look at. The problem here is arguably not on the Schematron side, but either in the CI/CD or in achieving composition more easily in the context of Schematron so as to avoid such a dependency.

The /tmp directory functionality is part of the shell scripting but then how paths get called from test files compiled down there to isolate them from commits, when the same thing is run in the GHA CI/CD remotely, is part of the issue. But it should be fixed on either counts. Tell you what, I can update this as part of the needed work that has to go into #25 and we close this out?

[wendellpiez]

Could be as simple as adding the file to a list of resources to be mirrored or copied....

[aj-stein-nist]

Could be as simple as adding the file to a list of resources to be mirrored or copied....

The resulting compiled XSLT will need an adjusted import statement at runtime and that can change dynamically, that is what I was trying to show in the screenshot (even if /tmp or /tmp.XXX12312312 a la mktemp for isolation of multiple passes in GHA) the source Schematron and other XSLTs for other kinds of tests don't know about that and xsl:import directives will fail for this reason.

If you want, we can discuss more, but it is likely in need of an overhaul, as we agreed to do and I committed to in #25. Discussing soon here or transferring the convo to #25, or holding off until we can discuss before making a decision, I'm fine either way.

[wendellpiez]

Would you be willing to consider the Schematron depending on an XSLT import reaching over the Internet?

In oXygen, such a resource could be cached locally and mapped with an XML Catalog for proxying, so the Internet dependency would not be absolute for the Schematron. (Presumably its own dependencies, i.e. pipeline stylesheets, would also be cached and proxied in such a scenario.)

Another way to put it is this: assuming a Schematron that must call a function that invokes a series of XSLTs (i.e. it can't readily be defined inline), what is the best way to support that function, under CI/CD?

Happy to be instructed on the mysteries of CI/CD.

[wendellpiez]

@aj-stein-nist could I bother you to reassess this for currency?

I bet you if there is still a problem, we could address it if we looked together.

If the Issue is now a nothing-to-do feel free to close it. Thanks!

[wendellpiez]

Please reopen or recreate as needed.