Pipfile needs a way to manually specify package installation order

[caspervdw]

GDAL is a package that is used by many GIS projects. During installation, GDAL's python bindings are compiled using the headers from the existing numpy version. So you need to install numpy before GDAL, else numpy bindings will not work.

This seems to be a use case for pipenv install --sequential, but I am not able to control the order of the sequential install.

NB: you need to have libgdal-dev on Ubuntu 16 and CPLUS_INCLUDE_PATH=/usr/include/gdal and C_INCLUDE_PATH=/usr/include/gdal to reproduce this.

pipenv install numpy gdal==1.11.2 --sequential works as expected.

Then, pipenv --rm, pipenv install --deploy --sequential gives me the wrong sequence: numpy is installed after gdal, and my numpy bindings do not work.

The sequence seems to origin from the Pipfile.lock, which is alphabetic. gdal is before numpy in the alphabet, so at least installation fails in a reproducible way.

How should I approach this issue?

Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
numpy = "*"
gdal = "==1.11.2"

[dev-packages]

[requires]
python_version = "2.7"

Pipfile.lock:

{
    "_meta": {
        "hash": {
            "sha256": "42b5387796abc5d9542f6213df851d564959e44cd1c1a0789fc9dc5e0e5f5d0f"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "2.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "gdal": {
            "hashes": [
                "sha256:7827197836690815b0142412a49a97c6c6a18b21a83ece8cd6d8ec66127f10df"
            ],
            "index": "pypi",
            "version": "==1.11.2"
        },
        "numpy": {
            "hashes": [
                "sha256:0074d42e2cc333800bd09996223d40ec52e3b1ec0a5cab05dacc09b662c4c1ae",
                "sha256:034717bfef517858abc79324820a702dc6cd063effb9baab86533e8a78670689",
                "sha256:0db6301324d0568089663ef2701ad90ebac0e975742c97460e89366692bd0563",
                "sha256:1864d005b2eb7598063e35c320787d87730d864f40d6410f768fe4ea20672016",
                "sha256:46ce8323ca9384814c7645298b8b627b7d04ce97d6948ef02da357b2389d6972",
                "sha256:510863d606c932b41d2209e4de6157ab3fdf52001d3e4ad351103176d33c4b8b",
                "sha256:560e23a12e7599be8e8b67621396c5bc687fd54b48b890adbc71bc5a67333f86",
                "sha256:57dc6c22d59054542600fce6fae2d1189b9c50bafc1aab32e55f7efcc84a6c46",
                "sha256:760550fdf9d8ec7da9c4402a4afe6e25c0f184ae132011676298a6b636660b45",
                "sha256:8670067685051b49d1f2f66e396488064299fefca199c7c80b6ba0c639fedc98",
                "sha256:9016692c7d390f9d378fc88b7a799dc9caa7eb938163dda5276d3f3d6f75debf",
                "sha256:98ff275f1b5907490d26b30b6ff111ecf2de0254f0ab08833d8fe61aa2068a00",
                "sha256:9ccf4d5c9139b1e985db915039baa0610a7e4a45090580065f8d8cb801b7422f",
                "sha256:a8dbab311d4259de5eeaa5b4e83f5f8545e4808f9144e84c0f424a6ee55a7b98",
                "sha256:aaef1bea636b6e552bbc5dae0ada87d4f6046359daaa97a05a013b0169620f27",
                "sha256:b8987e30d9a0eb6635df9705a75cf8c4a2835590244baecf210163343bc65176",
                "sha256:c3fe23df6fe0898e788581753da453f877350058c5982e85a8972feeecb15309",
                "sha256:c5eb7254cfc4bd7a4330ad7e1f65b98343836865338c57b0e25c661e41d5cfd9",
                "sha256:c80fcf9b38c7f4df666150069b04abbd2fe42ae640703a6e1f128cda83b552b7",
                "sha256:e33baf50f2f6b7153ddb973601a11df852697fba4c08b34a5e0f39f66f8120e1",
                "sha256:e8578a62a8eaf552b95d62f630bb5dd071243ba1302bbff3e55ac48588508736",
                "sha256:f22b3206f1c561dd9110b93d144c6aaa4a9a354e3b07ad36030df3ea92c5bb5b",
                "sha256:f39afab5769b3aaa786634b94b4a23ef3c150bdda044e8a32a3fc16ddafe803b"
            ],
            "index": "pypi",
            "version": "==1.14.3"
        }
    },
    "develop": {}
}

[techalchemy]

Oh that’s an interesting one!

Perhaps sequential installs should always start with Pipfile ordering and should always install top level packages first? /cc @uranusjr — do you think this impacts people who depend on things like Cython in their installation ordering?

[caspervdw]

I just need numpy before gdal. For larger projects, the ideal solution would be specifying just that instead of the slower sequential install.

Is there a way to extend extras=numpy for this use case?

[uranusjr]

It is probably best for package maintainers to have appropriate extras_require flags for those usages (e.g. gdal[numpy] to require numpy before gdal; I think we account for those already). But yes it would be nice if we can default to ordering in Pipfile, following the “not worst than the predecessor” (predecessor here being requirements.txt) rule.

[caspervdw]

@uranusjr That would be the ideal solution indeed. It seems that https://pypi.org/project/pygdal/ may solve the issue for me; they properly specify numpy as a dependency.

Apart from that, I would like to be able to control the --sequential order. To be complete: installation order now is alphabetic, and editable / VCS packages are installed first (but @techalchemy said earlier that a PR addressing the VCS issue is up already)

[simonk52]

(I've just stumbled across this same problem, thanks for the pointer to pygdal)

The way I see it, not all projects can be relied on to write their setup.py scripts in the "right" way. Therefore, it would be really nice to be able to override the dependencies of any package in the Pipfile. Somehow we would need to specify that, for the current application, gdal depends on numpy. Would something like this be possible?

[packages]
gdal = { setup_requires = ["numpy"] }

[OrangeDog]

Another use-case:
On Windows, python-ldap is rather tricky to build and they don't publish wheels to PyPi.
You need to be able to say that your wheel (e.g. downloaded from https://www.lfd.uci.edu/~gohlke/pythonlibs/) should be installed before anything that requires python-ldap.

[Eimis]

I assume this is not yet solved and when you run pipenv install it is still random ?

[uranusjr]

It is not random, but there is no way to customise ordering (I assume this is what you actually intend to ask).

[Eimis]

It is not random, but there is no way to customise ordering (I assume this is what you actually intend to ask).

Yes exactly. Thanks for the response, it's clear!

[mattskone]

This is not a proper solution, but it might be acceptable for some until (if) this feature is implemented.

pipenv install --dev appears to install dev-packages before the default packages, so after editing the Pipfile by placing the numpy dependency in dev-packages, and GDAL in packages, then pipenv install --dev --sequential seems to install numpy before GDAL and avoids the author's original error.

[matteius]

I believe this can be closed, more recent work on constraints from default and dev may have improved this situation. Feel free to open a new ticket for any new issues.