nixos/testing: add deprecation notice for Perl VM tests

[flokli]

Most VM tests have been migrated to use the python test driver
(introduced in #71684), the migration is tracked in #72828 (which also
thankfully uncovered and fixed many currently broken tests)

While increasing the acceptance and adoption of NixOS integration tests
by using a more popular language, there was also nobody willing to do
larger refactors in the currently very convoluted test infrastructure.

We plan to remove the perl infrastructure between the 20.03 and 20.09
release, to be able to do these refactorings.

Some people might be using Perl tests in their internal CI, so print a
warning for 20.03, and give users time to move to the python testing
infrastructure.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[jonringer]

I agree with the intent of this PR.

Just concerned that the warning message may get lost as nixos tests are particularly noisy. Using nix build it might be useful, but nix-build will definitely blanket the warning with noise.

[flokli]

@worldofpeace I assume you're adding a general explanation to the release notes as well?

[worldofpeace]

@worldofpeace I assume you're adding a general explanation to the release notes as well?

There's one thing that's being forgot here aside from the deprecation notice and it's nixosTest uses nixos/lib/testing.nix.

[worldofpeace]

@flokli I believe I will be adding that as a highlight.

[fpletz]

While in principle being in favour of this, I have a question about your refactoring plans: Will the current Python driver API (both Nix and Python) stay compatible? If we advocate to move away from the Perl test infra, we should have something ready that will still be compatible for at least the next release in half a year.

If there a big changes planned, keeping the Perl stuff around could be worthwhile and a manageable technical debt because both test drivers could be cleanly separated with the Perl driver in maintenance mode.

If the Python API is considered stable, awesome! 👍

[flokli]

I assume most refactorings will first happen behind the scenes, and we should be able to keep API compatibility for at least the next release. (There might be new features, but code assuming the current framework should still work) We might change subtest failure exit behaviour (#79328), but that won't affect overall test failure/success behaviour.

[worldofpeace]

This PR does update nixosTest to use the python driver, and you can't use the nixpkgs function to access the perl one anymore. @flokli noted in IRC that he didn't think that would be problematic.

[flokli]

Well, nixosTest seems to be quite newer the import ./make-test.nix pattern, which can still be used to execute perl tests.

I don't think it makes any sense to introduce a nixosTestPerl at least 😆

[worldofpeace]

Well, nixosTest seems to be quite newer the import ./make-test.nix pattern, which can still be used to execute perl tests.

I don't think it makes any sense to introduce a nixosTestPerl at least laughing

I think similarly that having anything like a nixosTestPerl would be pointless if it's just going to be removed in 20.09.