nixos: add python testing support

[tfc]

Motivation for this change

Myself and most people i work with don't know Perl but Python. NixOS integration tests are totally awesome and more people should use/write such to test software.

I would like to support the acceptance and adoption of NixOS integration tests by letting users choose the more popular language Python to implement the imperative integration test part.

Things done

I ported the test driver to python and forked the nix scripts. Of course the duplication can be reduced, but i did not want to put more effort into this in case this work is not accepted in this repo for some reason, so i am eager to hear/read your thoughts.

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

Notify maintainers

cc @edolstra @thefloweringash @abbradar @samueldr @srhb

Community discussion

After discussing this with some people at nixcon (including @edolstra @Lassulus @adisbladis @domenkozar @garbas ...) i arrived at the following TODO-list to make the whole thing merge-acceptable (Note that there was no decision):

• finish porting all funcitonality from the perl driver to python (ocr, coverage, ... it's not much missing)
• port all tests to python (so far these are roughly ~300 tests)
• remove perl test driver so there is only one
• squash all commits with good commit messages (following @jonringer's suggestion)

So far @blitz @jtraue @mdvjd are helping me already

[jonringer]

Do you mind changing the title and your first commiit to something like nixos: add python testing support

the later 3 commits according to contributing.md should be:

nixos/bittorent: convert test to python
nixos/login: convert test to python
nixos/postgres: convert test to python

[jonringer]

but this is awesome, i think python will enable a lower barrier-to-entry for nixos than Perl.

[tfc]

@jonringer I fixed the commit messages according to your suggestions.

[jonringer]

seeing as python support might be a disruptive change, please allow for some time for the necessary communication to happen :)

[blitz]

@tfc Super nice! Life is too short to learn Perl. :)

[jonringer]

cc @worldofpeace @FRidh

[FRidh]

I haven't written any NixOS tests myself, but is the language (Perl) such a barrier that we need to implement it (also) in Python? From what I understand the tests are often quite simple, not requiring any deep knowledge of Perl. Personally I don't mind having it in Python at all, but let's keep this open for a while so those that use the NixOS testing framework can review.

Also, it seems to be we need a code owner for the original parts and the newly added parts.

[tfc]

@FRidh perl is not a barrier to start writing simple nixos integration tests at all. Once you get used to them you would like to write fancier and fancier tests that use the language's facilities to reduce duplication and to make things a bit more dynamic, and maybe also add libraries (currently not supported in my proposal yet, but i also don't want to blow up the PR). That is where non-Perl users would like to use Python.

If an owner is needed for the Python part, i would volunteer as this would be heavily used by my colleagues at work.

[blitz]

Maybe an interesting data point: https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=%2Fm%2F05zrn

[worldofpeace]

TBH, I would play with testing much more strongly if it was in python.
I don't think it's too unreasonable to support multiple types of testing scripts.
Can we not have a testing-python.nix file? It's the same as testing.nix at a glance, but just changed to use the python things. So testing.nix needs support to use different testing drivers.

Though I'm realizing you've done this on purpose so as to get feedback.
Currently short on time, but I can give in depth thoughts and opinions soon.

[domenkozar]

cc @copumpkin you wanted this :D

[tfc]

Cc @adisbladis

[tfc]

Cc @azazel75

[jonringer]

@tfc I would hold off on converting a lot of the tests over to python. It will expand how much this PR is changing and make it harder to come to a decision on whether or not it should be merged.

Even though a lot of the converted tests seem to be a 1:1 translation :)

[worldofpeace]

After discussing this with some people at nixcon (including @edolstra @Lassulus @adisbladis @domenkozar @garbas ...) i arrived at the following TODO-list to make the whole thing merge-acceptable:

So it appears the people you've talked to at nixcon agreed on porting testing completely to python?
Not that I disagree with that, maintaining multiple testing drivers doesn't really sound all that great.

Can the people who discussed this at nixcon publicize their viewpoints here?
Because currently it seems there was an agreement to port testing completely to python and to accept this change, and maybe not everyone agrees with this, or at least the people @tfc had discussion with agreed with it. Mostly looking out for people so they can know why.

If my assumption here is incorrect, correction is welcome.

[flokli]

Thanks a lot for this!

[worldofpeace]

@flokli Why wasn't https://github.com/NixOS/nixpkgs/pull/71684/files#r340032396 done?
I believe I agreed with @edolstra here #71684 (comment). Is that going to be in another PR?

[flokli]

@worldofpeace there was no clear consensus on #71684 (comment), and as this can always be done in a follow-up PR, I didn't want to stall this any further.

[worldofpeace]

@flokli I think I linked the wrong comment https://github.com/NixOS/nixpkgs/pull/71684/files#r340031960

[flokli]

In that case, porting isn't finished, but I didn't consider this a requirement for this PR, which is why I asked about #71684 (comment).

I assume porting the remaining nixos vm tests will happen soon, and once that's done, we can remove the perl test (and maybe rename the python one to the original location).

Docs already only document the python one.

[FRidh]

Without going into whether Python is an improvement or not, but in these cases we should just write the data as JSON and read that, regardless of language. That means currently explicitly writing to JSON and in the future with structured attributes we can simply pass the dict as an attribute.

[domenkozar]

But... python does not look an improvement at all, it differs from perl only in syntax,

That's untrue, the major difference is that nowadays Python is on the top of the list of popular languages, so probably millions more are capable of contributing easily.

Even nodejs seems favorable than python.

That could be argued, but it's least designed language of the popular ones we have.

scala + monix,

Well scala is used in very specific settings, so that way I'd actually prefer perl since it's more approachable.

I think you're forgetting that any NixOS developer should be able to write these tests.

[FRidh]

@volth if you would like to discuss the future of the stdenv (language), please open a separate issue. I am sure others (including myself) would be interested in discussing it.

[domenkozar]

Quite a few scripts have been rewritten to python, so I'd rather say: we're replacing perl for python.

[domenkozar]

$ cd nixos
$ git co release-16.03
$ find . -name "*.py" | wc -l
1

$ find . -name "*.pl" | wc -l
8

$ git co master
$ find . -name "*.py" | wc -l
5

$ find . -name "*.pl" | wc -l
7

Where we have yet to remove test-driver.pl some day.

[aszlig]

Maybe it's a bit late, but just wanted to throw in my two cents on this (since I'm buried in other things lately :-/): I do like the switch, since I'm already using Python in a lot of my tests, particularly the WebDriver/Selenium ones, but I find the 4 space indentation and line length enforcement pretty awkward.

Maybe I'm not up-to-date on this (last checked was when writing the nginx-etag test) and it's no longer the case, but while I would prefer PEP-8 style indenting for standalone Python files, it's not only very weird to look at when used inline - given that Nix expressions commonly use 2 spaces for indentation - but also makes it annoying to tab twice in the editor for every block.

Same goes for the line length, I'm theoretically fine with the limit and I do enforce 79 chars myself when writing Nix code, but this falls short if you have things like this:

{ testScript = ''
    machine.succeed("ls ${someStorePath}")
  '';
}

Since black only knows about the resulting script, this would result in an error and you need to write it like this:

{ testScript = ''
    machine.succeed(
      "ls ${someStorePath}"
    )
  '';
}

... even though the line length is fine on the Nix side. Worse off, if you have a line where the derivation name is just long enough so black would break it, a simple version change of a derivation could break the validation of the test script, eg. if you have somedrv-1.0.99 versus somedrv-1.1.0.

[worldofpeace]

I have the same issues #72964 @aszlig

[tfc]

Same goes for the line length, I'm theoretically fine with the limit and I do enforce 79 chars myself when writing Nix code, but this falls short if you have things like this:
... even though the line length is fine on the Nix side. Worse off, if you have a line where the derivation name is just long enough so black would break it, a simple version change of a derivation could break the validation of the test script, eg. if you have somedrv-1.0.99 versus somedrv-1.1.0.

There is generally a flag to turn it off for tests.

https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/testing-python.nix#L99

I found black relatively "patronizing" during development, but in the end it gives all python code a "uniform" look and i find that pretty appealing while reading code.
But the problem you just described sounds indeed painful. If we ever run into it in a certain frequency, i could imagine relaxing specific black rules.

[Mic92]

Some tooling would help to automatically format nixos tests using black. This is not trivial though. Especially since strings could contain interpolations from nix.

[aszlig]

@tfc: Well as mentioned, I do like having an "uniform" look, but for inline code it find the disparity between the indentation levels pretty weird to look at. For the line length issue (or maybe even the indentation issue), I think #72964 would be a better place to discuss.

Addendum: I know that there is a flag to turn it off, but if I'd need to do that for every test contributed to nixpkgs, I wonder what benefits there are for the linter/formatter at all.