Provide a way for third-party test modules to be installed.

[gbicann]

Purpose

This PR adds a new method called install_custom_test_module() to Zonemaster::Engine::Test. It allows custom modules outside the Zonemaster::Engine::Test:: namespace to be added to Zonemaster as test modules. Currently the only way to do this is to patch the modules.txt file and intrude upon the Zonemaster namespace.

Context

I am developing a new Registry System Testing (RST) service that uses Zonemaster for DNS and DNSSEC testing. There are a handful of RST test cases that Zonemaster doesn't currently cover, which I don't think are appropriate for inclusion in the main Zonemaster test suite (for example, checking for the presence of valid IDNA2008 labels in SOA mname and rname fields, which is not something that most Zonemaster users will care about). However, I still want to use Zonemaster to run these test cases, and I think the ability to install custom modules may be more generally useful (for example, in future it might allow plugins for Zonemaster to be distributed separately, via CPAN).

Changes

There are two main changes:

1. the install_custom_test_module() method has been added to Zonemaster::Engine::Test. It accepts a class name in a string, eg:

Zonemaster::Engine::Test->install_custom_test_module( 'My::Module' );

It adds the module name to the internal module list, and updates the effective profile to add the module's test cases to the test sequence, checking to make sure that they don't collide with existing test case names.

2. the run_all_for(), run_module() and run_one() methods have been changed so that if a module name looks like a Perl class name (that is, it contains ::) it is used as-is rather than being prefixed with Zonemaster::Engine::Test::.

How to test this PR

This PR includes a test cast. Use prove t/custom.t to run it.

[matsduf]

@gbicann, thank you for your proposal. We will review this. Making Zonemaster more flexible is positive.

When you do some POC runs, do not forget to create a custom profile.json (e.g. in /etc/zonemaster) where the test cases are enabled for testing, or else they will not be run.

[gbicann]

@matsduf thanks for the feedback.

When you do some POC runs, do not forget to create a custom profile.json (e.g. in /etc/zonemaster) where the test cases are enabled for testing, or else they will not be run.

So install_custom_test_module() actually does this: it gets the effective profile, and appends the custom test cases to the list (checking to make sure they don'y collide with an existing case with the same name).

Would you prefer that patching the profile has to be done separately?

[matsduf]

I can see how you have done it, but I cannot see how you envision install_custom_test_module() to be run. I could be run by a custom client, but not by zonemaster-cli or Zonemaster-Backend.

How should Zonemaster know that there is a custom test module?

• One way would be for Zonemaster to check the Zonemaster/Engine/Test/ directory. If there is another Perl module try to use it as a additional test module, as a plug-in. This would not be fool-proof solution.
• Another option would be to add an entry in a custom profile.json in /etc/zonemaster

Your proposal would always run all test cases in a custom module. In standard Zonemaster only test cases listed in profile.json are run, which makes it possible to disable some test cases in a module, but not necessarily all.

[matsduf]

I like the idea of making it possible to add a plugin with addition custom test cases to zonemaster, but I am also curious about you example of additional test cases.

There are a handful of RST test cases that Zonemaster doesn't currently cover, which I don't think are appropriate for inclusion in the main Zonemaster test suite (for example, checking for the presence of valid IDNA2008 labels in SOA mname and rname fields, which is not something that most Zonemaster users will care about).

If you with your example mean that SOA MNAME must be a valid IDNA 2008 domain name and the same thing for the domain part of the SOA RNAME, I do not feel that those tests are alien to Zonemaster testing. I can envision including those.

[gbicann]

I cannot see how you envision install_custom_test_module() to be run. I could be run by a custom client, but not by zonemaster-cli or Zonemaster-Backend.

How should Zonemaster know that there is a custom test module?

On reflection, I agree that perhaps it would be better to add custom modules using an option in the profile. Perhaps a custom_modules property, like this:

{
    "custom_modules": [
      "My::Module"
    ]
}

Let me know what you think about that, I could probably rework this merge request to work that way. Please also let me know where documentation would need to be updated.

If you with your example mean that SOA MNAME must be a valid IDNA 2008 domain name and the same thing for the domain part of the SOA RNAME, I do not feel that those tests are alien to Zonemaster testing. I can envision including those.

That's good to hear. The full description of this test case is here. I have an implementation, so perhaps this could be dropped into Zonemaster::Engine::Test::Syntax, as syntax09?

[gbicann]

I've now implemented this approach in the custom-module-profile branch in my fork.

I am minded to close this PR and submit a new one with this approach instead. Let me know what you think.

[matsduf]

On reflection, I agree that perhaps it would be better to add custom modules using an option in the profile. Perhaps a custom_modules property, like this:

{
    "custom_modules": [
      "My::Module"
    ]
}

Let me know what you think about that, I could probably rework this merge request to work that way. Please also let me know where documentation would need to be updated.

We will discuss this in the work group.

That's good to hear. The full description of this test case is here. I have an implementation, so perhaps this could be dropped into Zonemaster::Engine::Test::Syntax, as syntax09?

I do not find the specification. Can you point exactly at the content?

Syntax09 is fine, but specification must be completed first, and the implementation must match the specification. We will review and possibly update the specification to match the "standards".

Your implementation can be sent in as a PR, but we must complete the specification before working on the implementation.

[matsduf]

@gbicann!

Creating a new test case, e.g. Syntax09, verifying that SOA MNAME and RNAME have valid IDN labels if any A-label or A-label like is found, is not very complicated. It should be done in two steps, first the specification is composed and approved, and then an implementation following the specification is created or finalized. If you have a start of a specification it would be helpful. You could provide that as an issue or PR to the zonemaster/zonemaster repository where it would belong.

You could also provide a list of other test cases that you think are missing today as in issue to zonemaster/zonemaster and we will look at that.

Adding flexibility to allow for a custom test module might require multiple changes in Zonemaster. We should discuss this in the Zonemeaster work group (Internetstiftelsen and Afnic) first to see what path we suggest. Or if we find severe blockers e.g. due to some magic in the code.

This would also create a new API in Zonemaster, i.e. between a custom module and Zonemaster-Engine. That API should be defined and documented so that custom modules are well-behaved and Zonemaster retain the interface needed for the custom module.

Nothing that I write here should be taken as a promise since this has to be discussed and prioritized in the Zonemaster project. What is the ICANN time line?

[mattias-p]

Thank you @gbicann for the feature request and for exploring the design space with your two drafts! We discussed this at our workgroup meeting yesterday and everyone agrees we want to see this feature implemented. We also had some ideas on what would be a fitting design. I promised to draw up a design proposal. I'll link to here in a day or two so you can review it.

[gbicann]

Hi @mattias-p and @matsduf, thank you so much for your time and attention on this, it's much appreciated. I look forward to seeing your design proposal. I hope my prototyping has been useful.

To answer Mats' question about ICANN's timeline: the new RST system needs to be ready by the end of 2024, however the additions to Zonemaster are a very small part of that project and do not block any other work so there is no particular urgency.

I'm interested to hear that you think there might be value in including our additional test cases into the main Zonemaster code base. We have four test cases in our plan which are as follows:

• dns-zz-idna2008-compliance - IDNA2008 compliance check
• dnssec-91 - Permitted signing algorithms
• dnssec-92 - Permitted DS record hash algorithm(s)
• dnssec-93 - NSEC3 iterations check

None of these are technical tests, they are checking to see that a registry operator has configured the TLD zone in accordance with ICANN policy. Therefore their applicability to other users of Zonemaster might be quite limited: this is why I proposed a way to load a custom test module, rather than just adding test cases to the existing test modules.

By the way, if we do end up implementing a custom test module, we'd almost certainly publish it under an open source license on CPAN, so you could always just add it as a dependency and incorporate it into Zonemaster that way :)

[matsduf]

@gbicann

I'm interested to hear that you think there might be value in including our additional test cases into the main Zonemaster code base. We have four test cases in our plan which are as follows:

• dns-zz-idna2008-compliance - IDNA2008 compliance check

Checking that SOA MNAME and RNAME fields have are valid IDN names, or just plain LDH names, could be integrated in Zonemaster. When it comes to RNAME is should be from label two, I guess. Checking all DNS records in apex is more complicated.

• dnssec-91 - Permitted signing algorithms

That should already be covered by DNSSEC05 and customize the levels for the message tags, shouldn't it?

• dnssec-92 - Permitted DS record hash algorithm(s)

That should already be covered by DNSSEC01 and customize the levels for the message tags, shouldn't it?

• dnssec-93 - NSEC3 iterations check

That should be covered by upcoming DNSSEC03 (still in develop branch), shouldn't it? It will soon be released and merged to master branch.

None of these are technical tests, they are checking to see that a registry operator has configured the TLD zone in accordance with ICANN policy. Therefore their applicability to other users of Zonemaster might be quite limited: this is why I proposed a way to load a custom test module, rather than just adding test cases to the existing test modules.

It will also open for the possibility to fork the standard test cases if changes of levels of message tags is not enogh.

[mattias-p]

Hello @gbicann, I'm sorry I took so long to get back to you. There was more nuance to this than I first anticipated, and of course other things got in between as well.

Anyways, I've made a new design for the custom test modules feature. It's expressed in a series of issues where each issue introduces one new idea and each idea builds on the previous one. Here they are in order:

1. Dynamically generated default profile file #1333
2. Define default message severities inside test modules #1334
3. Test module registration #1335
4. Split test_levels into multiple properties #1336
5. Custom test modules #1337

I based the new design on some of your ideas, applying them not only to the custom test modules but to the built-in ones as well to great effect. The scope of the new design is a bit larger than your drafts and the exact APIs and some of the mechanics had to change to work with other important use cases.

I would like to thank you for your efforts in drafting this feature. Your drafts greatly informed my understanding of your needs. I showed the new design to the other members of the Zonemaster work group and their initial reaction was very positive. If you want to review it from the perspective of your use case and let us know how it fits, I would appreciate that. But I do believe I have you covered.

If you want to implement any of this you are very welcome. Otherwise we'll do it within the work group. I'm not sure about our prioritization of the feature but I believe it's fairly high.

[gbicann]

Hi @mattias-p and @matsduf, apologies for the long delay in replying.

Our development team has made the decision that the additional test cases we need for RST v2.0 will be implemented in our Java-based test runner, since this is the preferred language that's used for most ICANN-internal projects, and is therefore easier for the Org to maintain.

However, speaking as a Perl hacker and regular user of Zonemaster, what you outlined above looks really good, and I am pretty sure it will be of use in the future, so I would like to offer my thanks for your efforts. It's greatly appreciated!