Add config file

[merwok]

I end up writing wrapper scripts to pass options (like no-header) and support multiple .in files (#532).

If pip-compile knew to find its options in .pip-tools.cfg or tox.ini (many Python development tools such as pytest, flake8, coverage can find their config in tox.ini, even when they’re not run inside tox), I would put the options there and tell my co-workers they can simply run pip-compile file.in.

Idea taken from pip-tools’ fork prequ.

[vphilippon]

I would suggest to go with a pip-tools.cfg file or something alike, rather than tox.ini. I feel like options in tox.ini should be about testing and such.

Nice suggestion 👍

[suutari-ai]

How about a [pip-tools] section in setup.cfg? Most Python tools can read their config from setup.cfg (e.g. pytest, flake8, isort, etc.) and IMHO it's nicer to have fewer files in the project root.

[vphilippon]

I haven't looked much at the setup.cfg file. If it's meant to be extensible for other tools like that, then that sounds more reasonable to me.

[merwok]

It’s not meant to (the section names are made to match setup.py commands), but some tools look there too.

A difference is that a tox.ini section like pytest needs to be tool:pytest in setup.cfg.

[johnthagen]

Perhaps the standardized pyproject.toml should be used?

[tool.pip-tools]
# ...

[elcolie]

Any progress on this so far?
Since I am using base.txt, local.txt, and production.txt in my repository. pip-compile is amazing
base.txt

pytz==2020.1  # https://github.com/stub42/pytz
python-slugify==4.0.1  # https://github.com/un33k/python-slugify
Pillow==7.2.0  # https://github.com/python-pillow/Pillow
argon2-cffi==20.1.0  # https://github.com/hynek/argon2_cffi
redis==3.5.3  # https://github.com/andymccurdy/redis-py
hiredis==1.1.0  # https://github.com/redis/hiredis-py
celery==4.4.7  # pyup: < 5.0  # https://github.com/celery/celery
django-celery-beat==2.0.0  # https://github.com/celery/django-celery-beat
flower==0.9.5  # https://github.com/mher/flower
uvicorn==0.11.8  # https://github.com/encode/uvicorn
django-extensions==3.0.7
arrow==0.16.0
tqdm==4.48.2
django-filter==2.3.0

# Django
# ------------------------------------------------------------------------------
django==3.1.1  # pyup: < 3.1  # https://www.djangoproject.com/
django-environ==0.4.5  # https://github.com/joke2k/django-environ
django-model-utils==4.0.0  # https://github.com/jazzband/django-model-utils
django-allauth==0.42.0  # https://github.com/pennersr/django-allauth
django-crispy-forms==1.9.2  # https://github.com/django-crispy-forms/django-crispy-forms
django-redis==4.12.1  # https://github.com/jazzband/django-redis
# Django REST Framework
djangorestframework==3.11.1  # https://github.com/encode/django-rest-framework
django-cors-headers==3.5.0 # https://github.com/adamchainz/django-cors-headers

local.txt

-r base.txt

Werkzeug==1.0.1 # https://github.com/pallets/werkzeug
ipdb==0.13.3  # https://github.com/gotcha/ipdb
psycopg2==2.8.5 --no-binary psycopg2  # https://github.com/psycopg/psycopg2
watchgod==0.6  # https://github.com/samuelcolvin/watchgod

# Testing
# ------------------------------------------------------------------------------
mypy==0.770  # https://github.com/python/mypy
django-stubs==1.5.0  # https://github.com/typeddjango/django-stubs
pytest==6.0.1  # https://github.com/pytest-dev/pytest
pytest-sugar==0.9.4  # https://github.com/Frozenball/pytest-sugar

# Documentation
# ------------------------------------------------------------------------------
sphinx==3.2.1  # https://github.com/sphinx-doc/sphinx
sphinx-autobuild==2020.9.1 # https://github.com/GaretJax/sphinx-autobuild

# Code quality
# ------------------------------------------------------------------------------
flake8==3.8.3  # https://github.com/PyCQA/flake8
flake8-isort==4.0.0  # https://github.com/gforcada/flake8-isort
coverage==5.2.1  # https://github.com/nedbat/coveragepy
black==20.8b1  # https://github.com/ambv/black
pylint-django==2.3.0  # https://github.com/PyCQA/pylint-django
pylint-celery==0.3  # https://github.com/PyCQA/pylint-celery
pre-commit==2.7.1  # https://github.com/pre-commit/pre-commit

# Django
# ------------------------------------------------------------------------------
factory-boy==3.0.1  # https://github.com/FactoryBoy/factory_boy

django-debug-toolbar==2.2  # https://github.com/jazzband/django-debug-toolbar
django-extensions==3.0.7  # https://github.com/django-extensions/django-extensions
django-coverage-plugin==1.8.0  # https://github.com/nedbat/django_coverage_plugin
pytest-django==3.9.0  # https://github.com/pytest-dev/pytest-django

production.txt

# PRECAUTION: avoid production dependencies that aren't in development

-r base.txt

gunicorn==20.0.4  # https://github.com/benoitc/gunicorn
psycopg2==2.8.5 --no-binary psycopg2  # https://github.com/psycopg/psycopg2
Collectfast==2.2.0  # https://github.com/antonagestam/collectfast
sentry-sdk==0.17.3  # https://github.com/getsentry/sentry-python

# Django
# ------------------------------------------------------------------------------
django-storages[boto3]==1.10  # https://github.com/jschneier/django-storages
django-anymail[mailgun]==7.2.1  # https://github.com/anymail/django-anymail

[jcushman]

If anyone wants to take this on, I think a really good option would be to copy the code from the black package, which reads config options from pyproject.toml and uses the click library to parse arguments the same as pip-tools. It's just this click setting and adapting the read_pyproject_toml function and its support functions to pip-tools:

@click.option(
    "--config",
    type=click.Path(
        exists=True,
        file_okay=True,
        dir_okay=False,
        readable=True,
        allow_dash=False,
        path_type=str,
    ),
    is_eager=True,
    callback=read_pyproject_toml,
    help="Read configuration from FILE path.",
)

(I would remove the find_user_pyproject_toml() stuff from black's implementation, since I think having a user-level settings file is less useful for pip-tools than for black.)

[AndydeCleyre]

• I don't love the idea of options that I don't specify on the command line being inserted on my behalf when I run pip-compile
• I do recognize the need to wrap pip-compile commands in higher level, configured functions (hence my zpy project), both for personal use, and for dev team consistency.

IMO a function or script is better for this case than a configuration, as what we need to get done is often an ordered sequence of commands, not just specific options to a single invocation.

So for standardizing the operations across a team, I would either use a shell script, a Makefile (actually I wouldn't but if you like Makefiles go for it), or some invocation helper such as invoke, taskipy, poethepoet, pypyr, or doit (taskipy and poethepoet being most shell-command-friendly I think, and with definitions inside pyproject.toml).

EDIT: or nox

[netrix]

I think it would really help with (IMO) no inconsistency to add support for configuration in pyproject.toml. One of such use cases is a possibility to specify --extra-index-url or just different PyPI server that is for now impossible in pip-compile in any other way than specifying CLI parameter. Unfortunately for some use cases (creating an application with no package) PEP-621 that defines the project configuration doesn't take it under consideration.

In projects like poetry or pdm that support dependency locking but do way more than one thing there are sections like [[tool.poetry.source]], [[tool.pdm.source]] while pip-compile has only CLI parameter.

pip-compile actually supports CLI parameters in some way (maybe indirectly but you can add --extra-index-url in requirements.in) but pyproject.toml is discriminated here. It can read dependencies but to support different PyPI servers it can only be configured with CLI.

[lordmauve]

for standardizing the operations across a team, I would either use a shell script, a Makefile, or some invocation helper

The problem we face with doing something like this is fragmentation. If my org scales to 2000 repos (as I estimate it will) I don't want to have to maintain 2000 different noxfile.py scripts.

It's much more convenient to update settings in pyproject.toml - I can automate that with tomlkit, and we have to do this for tools like mypy and black already.

[AndydeCleyre]

FWIW you can use taskipy or poethepoet, where the task definitions exist inside pyproject.toml.

[lordmauve]

Those are probably better than a completely free-form approach like nox but they still offer arbitrary command invocation like shell syntax.

I'd like completely declarative configuration that I can statically analyse, validate and upgrade, no imperative code or command lines.

[AndydeCleyre]

What do you think about, rather than specifying a single set of always-on options, the pyproject.toml could contain any number of named pip-tools "profiles" (sets of options), which would be used by explicitly passing something like --profile NAME?

And any other passed arguments would override individual options in the profile?

[webknjaz]

FWIW, I'd also like a separate config file. Using a single config for randomly smashed sections from different tools is really terrible for maintaining configs through multiple projects.

[AndydeCleyre]

How does this sound?

Three possible config files

• if specified with an option during invocation, use that
• otherwise, check for an independent config file in the current folder
• if not found, check inside pyproject.toml in the current folder

If that sounds good, decisions remaining include:

• naming the option (e.g. --config, --config-file)
• naming the independent file (e.g. pip-tools.toml, .pip-tools.toml)
• structuring the config (named profiles, or just one set of options?)
  • this potentially leads to more CLI options (e.g. --profile)

I don't love toml but it's the natural choice here.

[j00bar]

Took a stab at a PR for this. My team uses pip-tools and AWS CodeArtifact - when we pip-compile our requirements, the temporary access token from CodeArtifact is by default included in the index-url written to requirements.txt and we'd really love to be able to set emit-index-url = false in a pyproject.toml and be done with it. So I'm motivated to do what it takes to get this to where it's deemed mergeable. Thanks!

[AndydeCleyre]

@j00bar Do you have any thoughts regarding the points and questions in my previous comment?

[j00bar]

@j00bar Do you have any thoughts regarding the points and questions in my previous comment?

Sure.

• The PR lets you specify an alternate TOML file explicitly.
• As far as a pip-tools specific TOML, I'm ambivalent. I'm observing the general trend being tools trying to convene around pyproject.toml, which I think is a good thing (sorry, but still hi @webknjaz long time no see), and other repo droppings are really for supporting older legacy behavior which we don't have here.
• I personally think "profiles" is overkill but could be achieved by making different TOML files for each profile and specifying them explicitly, which would be the same amount of effort for the developer.

I chose this approach to try for "make simple things simple and complex things possible".

[webknjaz]

@j00bar I'll be insisting on having an option to have a section in a dedicated config so that on-scale FOSS maintenance isn't insufferable...
I don't think that adding a config filename selection logic in a single place would complicate the implementation.

[felipecwb]

is this config file option released?
I tried to setup with version 6.13.0 and didn't work.

[j00bar]

The PR is labeled that it will be included in the 6.14.0 release.