Add pyproject.toml to dev-setup-ractoring

[afuetterer]

Hi there,

update: This PR got a little bit out of hand and I added a few more suggestions. Please let me know what we think.

My goal was to bring down the durations of your CI runs, which I succesfully did.

CI duration

I saw that your test suite has an impressive number of > 17k tests. The runtime of a single test run (one Python version + one DB) on CI took between 1 and 2 hours. So enough time for coffee and lunch. The PR proposes a few performance improvement in your test suite as well your CI setup.

Proposed changes (succesfully tested)

• update used GitHub actions
• set up pip cache in CI
• separate DB setup steps in CI (i.e. do not start mysql service when testing postgres)
• use the installed postgresql server in ubuntu 20.04 image in CI (i.e. no docker service), sadly I could not figure out how to access the postgresql db with postgres/postgres credentials, so I created a new user (postgres_user)
• update testing dependencies (pytest, pytest-cov, etc.)
• remove obsolete package: pytest-pythonpath
• run tests in parallel using pytest-xdist
• run tests in random order to identify dependent test cases
• use faster password hasher when testing, i.e. MD5PasswordHasher
• set DEBUG=False and other suggestions in testing config

So far all tests are still working and the run time came done to ~ 15 minutes per run. This is still enough time for coffee, but not for lunch anymore. I think you could do even better with more tweaks. But please let me know what you think.

For example: Do you really need to run the tests against sqlite in CI? Actually you test Django's capability to handle different backends, right?

Unable (unsuccesfully tested)

• apt package installation takes ~ 1 minute, there is a GitHub action to cache apt package installation, but it does not work with latex packages

TOML setup

There is still room for improvement, but I wanted to suggest the toml for further discussion.

I added the following dependency groups:

• mysql
• postgres
• tests
• dev
• ci

But tests, dev and ci are pretty similar.

Do you want to enforce using pre-commit for contributors? That is always a good idea. But if you want to use isort, flake8, (maybe black), you need to have on linting/reformatting-PR, I guess with the described 600+ files being changed. Also pre-commit should run on CI.

Flake8 sadly cannot be configured using the toml file. I would nowadays always recommend using ruff, which is really great and fast and adopted by major projects like pandas. ruff can replace isort and flake8 in my opinion.

Another question is still #648. The installation process using the toml file does not work with Python 3.6.

What do you think?

[jochenklar]

How big is the difference between md5 hashing and the regular PBKDF2? If we change the settings I would also change the settings in my dev setup (which is documented here: https://github.com/rdmorganiser/rdmo/blob/master/docs/dev.md). The idea is to use the same fixtures for development as for testing.

[afuetterer]

This check can be set as required in pull requests.

[jochenklar]

Yes, thanks, then I will just update my dev setup here when I use the fixtures in testing.

[jochenklar]

Hmm, https://pypi.org/project/django-plaintext-password/ 💣

[afuetterer]

Hmm, https://pypi.org/project/django-plaintext-password/ 💣

You want to use this instead to speed up things even more?

[jochenklar]

I tried the plaintext stuff and luckily it is not faster than md5. It it probably about the rotations and not about the algorithm.

time pytest --reuse-db rdmo/conditions

master:
real    0m27.273s
user    0m22.530s
sys 0m1.102s

md5:
real    0m20.340s
user    0m15.641s
sys 0m1.167s

plain:
real    0m20.881s
user    0m16.081s
sys 0m1.141s

[jochenklar]

The other new settings don't seem to make anything faster (for rdmo/conditions). I will try with the whole code.

[jochenklar]

time pytest --reuse-db rdmo/projects

master:
real    40m21.091s
user    34m7.364s
sys 1m20.997s

md5:
real    16m28.026s
user    10m15.002s
sys 1m38.042s

even more impressive, this is only the md5 hasher. I am checking the other options, but I think we can remove them (except the DEBUG settings). We could also put the logging part in the mysql/postgres/sqlite files so that they are still used locally.

[MyPyDavid]

How big is the difference between md5 hashing and the regular PBKDF2? If we change the settings I would also change the settings in my dev setup (which is documented here: https://github.com/rdmorganiser/rdmo/blob/master/docs/dev.md). The idea is to use the same fixtures for development as for testing.

I think that the password fixtures for development are now broken. Should this hashing algorithm also be part of the rdmo-app settings for debug?
Including this in the rdmo-app settings fixes it

PASSWORD_HASHERS = ("django.contrib.auth.hashers.MD5PasswordHasher",)