Test conversion coverage

[strogonoff]

We’d want to at least know that all data from the source that should be rendered, is rendered.

Data can be missing at two points:

• Not extracted from the source by source adapter

  These are hard to detect, if a bug causes some resource to not be extracted, and all references to it as well. Then it would go undetected.

• Not rendered as content by content adapter

  Some of these are already caught currently (for now just logged): when resource A references resource B, but resource B doesn’t get included in the content, this makes A -> B a broken reference, which Anafero notes. However, there still can be a combo of bugs that causes some resources to not be rendered, and all references to them as well, which would make it undetectable.

  If Anafero could just require that all resources obtained from the store appear in site content, then it could be possible to address this level fully automatically. It’s unclear whether we can require this. Presumably, it may be valid that some content adapters may choose to only render some resources? To be determined.

There are two abstraction levels coverage can be tested:

1. Regardless of whatever Anafero does, Firelight can have its own integration test that checks that Anafero-generated HTML covers source XML. This logic can be Metanorma-specific. While we work with Presentation XML, it’s probably enough to just check that all strings appear, minus irrelevant contents (e.g., semantic tags, localized strings, and so on).
2. Anafero can implement generic (not MN specific) built-in validation for store adapters and possibly (see above) content adapters. It could then run them on every build and automatically report coverage issues, in addition to already reported broken references.
   • If adapter transforms was declaratively defined, I think it could be possible to almost guarantee integrity. However, it’s not the case, adapters can imperatively transform source data while converting data, so it’s impossible to do fully automatically.
   • If declarative definition of store adapters is infeasible at this point (not impossible but difficult, especially with store adapters that work with arbitrary data, in theory even binary blobs), Anafero could allow or require adapters to provide validation functions for this purpose, making it the responsibility of adapter author to define a checker of some sort but taking care of running it during build. In this case, there is always a chance validation functions will be out of sync with adapter implementation, causing frustrating developer experience. Good error reporting could help, however.

We might start with 1 and later switch to 2. In 2, store adapters will be the focus, since reference checking should cover much of content adapter output.