Updating and adding all dependencies of future openstack sdk modules pull request

[costrouc]

Motivation for this change

I am trying to package the openstack python modules. oslo.config, oslo.db, and openstacksdk have ALOT of dependencies that are not currently packaged in nix. Along the way I have move all dependencies that were in python-packages.nix to python-modules/..... There are a lot of package in this commit (sorry and it there is any what that I can make this easier please let me know). This set of commits prepares all the dependencies for openstack modules.

I followed best practices. Testing was always used unless it created a circular dependency or was not packaged in the pypi repository. Otherwise tests were added to the package if it did not exist.

To help with code review here all all the package versions that need to be built with GrahmcOfBorg.

python27Packages.pathlib2
python27Packages.ddt
python27Packages.testrepository
python27Packages.enum34
python27Packages.eventlet
python27Packages.ipaddress
python27Packages.jmespath
python27Packages.mock
python27Packages.monotonic
python27Packages.munch
python27Packages.netaddr
python27Packages.netifaces
python27Packages.nose
python27Packages.oauthlib
python27Packages.warlock
python27Packages.fasteners
python27Packages.mox3
python27Packages.traceback2
python27Packages.linecache2
python27Packages.pep8
python27Packages.testscenarios
python27Packages.paramiko
python27Packages.prettytable
python27Packages.pyinotify
python27Packages.subunit
python27Packages.xattr
python27Packages.unittest2
python27Packages.unicodecsv
python27Packages.python-json-logger
python27Packages.daiquiri
python27Packages.python-coveralls
python27Packages.flake8-polyfill
python27Packages.flake8-docstrings
python27Packages.runtest
python27Packages.hacking
python27Packages.reno
python27Packages.cliff
python27Packages.debtcollector
python27Packages.stestr

python36Packages.pathlib2
python36Packages.ddt
python36Packages.testrepository
python36Packages.enum34
python36Packages.eventlet
python36Packages.ipaddress
python36Packages.jmespath
python36Packages.mock
python36Packages.monotonic
python36Packages.munch
python36Packages.netaddr
python36Packages.netifaces
python36Packages.nose
python36Packages.oauthlib
python36Packages.warlock
python36Packages.fasteners
python36Packages.mox3
python36Packages.traceback2
python36Packages.linecache2
python36Packages.pep8
python36Packages.testscenarios
python36Packages.paramiko
python36Packages.prettytable
python36Packages.pyinotify
python36Packages.subunit
python36Packages.xattr
python36Packages.unittest2
python36Packages.unicodecsv
python36Packages.python-json-logger
python36Packages.daiquiri
python36Packages.python-coveralls
python36Packages.flake8-polyfill
python36Packages.flake8-docstrings
python36Packages.runtest
python36Packages.hacking
python36Packages.reno
python36Packages.cliff
python36Packages.debtcollector
python36Packages.stestr

Things done

See individual commits for more detail.

pythonPackages.pathlib2: refactor add testing all pass
pythonPackages.ddt: 1.0.0 -> 1.2.0
pythonPackages.testrepository: refactor move to python-modules
pythonPackages.enum34: refactor move to python-modules
pythonPackages.eventlet: 0.20.0 -> 0.24.1
pythonPackages.ipaddress: 1.0.18 -> 1.0.22
pythonPackages.jmespath: 0.9.0 -> 0.9.3
pythonPackages.mock: refactor
pythonPackages.monotonic: 1.3 -> 1.5
pythonPackages.munch: refactor move to python-modules
pythonPackages.netaddr: refactor
pythonPackages.netifaces: 0.10.6 -> 0.10.7
pythonPackages.nose: refactor add meta move to python-modules
pythonPackages.oauthlib: 2.0.0 -> 2.1.0
pythonPackages.warlock: 1.2.0 -> 1.3.0
pythonPackages.fasteners: refactor move to python-modules
pythonPackages.mox3: 0.23.0 -> 0.26.0
pythonPackages.traceback2: refactor move to python-modules
pythonPackages.linecache2: refactor move to python-modules
pythonPackages.pep8: refactor move to python-modules
pythonPackages.testscenarios: 0.4 -> 0.5.0
pythonPackages.paramiko: refactor move to python-modules
pythonPackages.prettytable: 0.7.1 -> 0.7.2
pythonPackages.pyinotify: refactor move to python-modules
pythonPackages.subunit: 1.0.0 -> 1.3.0
pythonPackages.xattr: 0.7.8 -> 0.9.6
pythonPackages.unittest2: refactor move to python-modules
pythonPackages.unicodecsv: refactor move to python-modules
pythonPackages.python-json-logger: init at 0.1.9
pythonPackages.daiquiri: init at 1.5.0
pythonPackages.python-coveralls: init at 2.9.1
pythonPackages.flake8-polyfill: init at 1.0.2
pythonPackages.flake8-docstrings: init at 1.3.0
pythonPackages.runtest: init at 2.1.0
pythonPackages.hacking: init at 1.1.0
pythonPackages.reno: init at 2.10.0
pythonPackages.cliff: init at 2.13.0
pythonPackages.debtcollector: init at 1.20.0
pythonPackages.stestr: init at 2.1.1

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Fits CONTRIBUTING.md.

---

[xeji]

This causes a mass rebuild and needs to go to staging.

[costrouc]

Does this require me to pull request to a different branch? Or is this done on the other side?

[xeji]

Does this require me to pull request to a different branch?

Yes, to the staging branch. But you can modify this existing PR in two steps:

1. modify the PR's base branch to staging using the Edit button next to PR title. (after this, the PR may temporarily show lots of unwanted commits, but these disappear after step 2).
2. git rebase your local branch on staging and force-push to github

[xeji]

You could also split this PR into two parts: one part (all the new packages plus those changes that don't cause mass rebuilds) goes against master, the second part (changes to existing packages that cause mass rebuilds - probably mock, paramiko, unittest etc.) goes to staging. Up to you.
Things that go through staging can take 1-2 weeks to end up in master.

[costrouc]

I am in no rush to get this into master so 1-2 weeks is fine. Just a side project I've been working on the package openstack. Thanks so much for your help!

Question relevant to this. From what I have gathered the long term goal with python is to move all of the packages into python-modules. python-packages.nix still has many packages that have not been moved to python-modules/..... Many of these packages when moved will have to rebuild because they are using fetchurl instead of fetchPypi and are not updated to the current version. So massive rebuilds are inevitable for all of these packages right?

[xeji]

So massive rebuilds are inevitable for all of these packages right?

That obviously depends on the number of packages depending on them 😄 . But you're right, changing a larger number of python packages will often result in a mass rebuild. That's why we try to bundle updates into large PRs (example: #43972) once in a while.

[FRidh]

Related discussion on Discourse https://discourse.nixos.org/t/how-to-handle-packaging-python-packages-that-pin-dependency-version/918/8

[costrouc]

After having that discussion with you @FRidh on discourse I understand that supporting packages that are complex and pin dependencies are complicated and make the work on maintainers much harder. So I am not sure how much I want to persue this pull request. While I did spend a lot of time on the pull request I do not think that supporting openstack modules is a great path.

I am thinking of just splitting this up into smaller pull requests and not adding packages that are needed for openstack.

[FRidh]

@costrouc add those that you think are useful and good to have and that you think you can and want to maintain.

[Ekleog]

(triage) @costrouc From my reading of your last message, I think this PR could be closed until you add only the packages you need? :)

[Valodim]

For the record, and those googling "nixpkgs openstacksdk": openstacksdk can at this point be packaged with trivial packagings of os-service-types and keystoneauth1, albeit without checks.

Simple gist of the package: https://gist.github.com/Valodim/4dd4d052f6f1124afcd1729b3d28c5a5

The checks of those packages have circular dependencies (os-service-types (test) -> keystoneauth1 -> os-service-types), and also quite a bunch of missing deps. I believe this is the original reason this PR required so many added packages, so it's still true that this is overall hard to package properly. I'm not sure we can upstream this into nixpkgs easily, unless we forego all tests.