Git submodules in docker

[peter-ki]

pipenv, version 11.10.0

I have a submodule ./dep1 in my git repo and am adding it to pipenv via pipenv install -e dep1, which works just fine and adds the line

"abc123" = {path = "./dep1", editable = true}

to my Pipfile. Then I use a Dockerfile as follows:

FROM kennethreitz/pipenv
COPY ./app /app
COPY ./dep1 /app/dep1
CMD python3 start.py

where I added the third line, so the submodule is available in the WORKDIR. However, upon docker build I am getting the following error

+ pipenv install --deploy --system
Installing -e ./dep1…

Error:  An error occurred while installing -e ./dep1!
./dep1 should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+

This is unexpected, since ./dep1 is a path to a local project, relative to the WORKDIR in the docker image with the Pipfile, which is /app. Am I missing something or is this a bug?

[erinxocon]

Hi @peter-ki! Thanks for using pipenv! I need to update the docker instance to use the most current version of pipenv. Can you try rewriting your dockerfile to pull from python:3.6 or whichever version you are using and then pip install --upgrade pipenv? I'm wondering if our latest release fixes this issue. Additionally could you post your pipfile and Dockerfile? Also would it be possible to launch an interactive docker session and run the commands requested by the issue template so I can see the output? I know that's a lot, docker issues can be problematic to troubleshoot.

[erinxocon]

The dockerfile would be closer to ours in the repo:

FROM ubuntu:18.04

# -- Install Pipenv:
RUN apt update && apt install python3-pip git -y && pip3 install pipenv

ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8

# -- Install Application into container:
RUN set -ex && mkdir /app

WORKDIR /app

# -- Adding Pipfiles
ONBUILD COPY Pipfile Pipfile
ONBUILD COPY Pipfile.lock Pipfile.lock

# -- Install dependencies:
ONBUILD RUN set -ex && pipenv install --deploy --system

[peter-ki]

@erinxocon thank you very much for the fast response!

Interesting. I tried to work with the Dockerfile you provided, which seems to take the process a step further. I get a different error, though:

+ pipenv install --deploy --system
Installing -e file:///app/data-cache...
Obtaining file:///app/data-cache
Installing collected packages: DataCache
  Running setup.py develop for DataCache
Successfully installed DataCache

Adding -e file:///app/data-cache to Pipfile's [packages]...
Your Pipfile.lock (34a2ab) is out of date. Expected: (6bd758).
Aborting deploy.

The Pipfile is

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
kafka = "*"
pandas = "*"
numpy = "*"
"1bc1e69" = {path = "./data-cache", editable = true}

[dev-packages]

[requires]
python_version = "3.6"

and the Dockerfile now is

FROM ubuntu:18.04

# -- Install Pipenv:
RUN apt update && apt install python3-pip git -y && pip3 install pipenv

ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8

# -- Install Application into container:
RUN set -ex && mkdir /app

COPY ./services /app
COPY ./data-cache /app/data-cache

WORKDIR /app

# -- Adding Pipfiles
COPY Pipfile Pipfile
COPY Pipfile.lock Pipfile.lock

# -- Install dependencies:
RUN set -ex && pipenv install --deploy --system

CMD python3 start.py

Hm...

I also could run in the container interactively. But what does commands requested by the issue template refer to?

[erinxocon]

@peter-ki Thanks for the info. So when you create a bug report we ask you to run the command python -m pipenv.help and paste the output, it contains diagnostic info for use in troubleshooting the problem. And since yours is with a local package it would be helpful to get a capture of that. So if you could run python -m pipenv.help in the interactive session and paste the output that would be appreciated!

[erinxocon]

@peter-ki Also what vesion of pipenv are you using to generate the lockfile? Can you try updating to our latest release using pip/pip3 install pipenv --upgrade and regenerate the lockfile and then rebuild the docker container with the new lockfile? The error is telling you that your lockfile is out of date and can't be used with --deploy

[peter-ki]

Ah sorry, got it. Here you go:

root@36f1e613ddbd:/app# python3 -m pipenv.help

$ python -m pipenv.help output

Pipenv version: '2018.6.25'

Pipenv location: '/usr/local/lib/python3.6/dist-packages/pipenv'

Python location: '/usr/bin/python3'

Other Python installations in PATH:

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 3.6.5: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.9.87-linuxkit-aufs',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Mar 14 15:12:16 UTC 2018',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• LC_ALL
• LS_COLORS
• LESSCLOSE
• LANG
• KAFKA_SERVER_ENDPOINTS
• HOSTNAME
• PWD
• HOME
• TERM
• SHLVL
• PATH
• LESSOPEN
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
• LANG: C.UTF-8
• PWD: /app

---

Contents of Pipfile ('/app/Pipfile'):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
kafka = "*"
pandas = "*"
numpy = "*"
"1bc1e69" = {path = "./data-cache", editable = true}
datacache = {editable = true, file = "file:///app/data-cache"}

[dev-packages]

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/app/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "599ae093fa031e777034f7217b53754baeca64542ef99125d2603a6b2834a2ab"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "1bc1e69": {
            "editable": true,
            "path": "./data-cache"
        },
        "kafka": {
            "hashes": [
                "sha256:35a49e9a432f6792aad48b2ab09d03867357aaa06eacba2678b7fd0808156e5c",
                "sha256:ffb3348cd918c31344b1a2eb61f692cbc423c83b1b06466a86de3db8a3f6a018"
            ],
            "index": "pypi",
            "version": "==1.3.5"
        },
        "numpy": {
            "hashes": [
                "sha256:07379fe0b450f6fd6e5934a9bc015025bb4ce1c8fbed3ca8bef29328b1bc9570",
                "sha256:085afac75bbc97a096744fcfc97a4b321c5a87220286811e85089ae04885acdd",
                "sha256:2d6481c6bdab1c75affc0fc71eb1bd4b3ecef620d06f2f60c3f00521d54be04f",
                "sha256:2df854df882d322d5c23087a4959e145b953dfff2abe1774fec4f639ac2f3160",
                "sha256:381ad13c30cd1d0b2f3da8a0c1a4aa697487e8bb0e9e0cbeb7439776bcb645f8",
                "sha256:385f1ce46e08676505b692bfde918c1e0b350963a15ef52d77691c2cf0f5dbf6",
                "sha256:4d278c2261be6423c5e63d8f0ceb1b0c6db3ff83f2906f4b860db6ae99ca1bb5",
                "sha256:51c5dcb51cf88b34b7d04c15f600b07c6ccbb73a089a38af2ab83c02862318da",
                "sha256:589336ba5199c8061239cf446ee2f2f1fcc0c68e8531ee1382b6fc0c66b2d388",
                "sha256:5edf1acc827ed139086af95ce4449b7b664f57a8c29eb755411a634be280d9f2",
                "sha256:6b82b81c6b3b70ed40bc6d0b71222ebfcd6b6c04a6e7945a936e514b9113d5a3",
                "sha256:6c57f973218b776195d0356e556ec932698f3a563e2f640cfca7020086383f50",
                "sha256:758d1091a501fd2d75034e55e7e98bfd1370dc089160845c242db1c760d944d9",
                "sha256:8622db292b766719810e0cb0f62ef6141e15fe32b04e4eb2959888319e59336b",
                "sha256:8b8dcfcd630f1981f0f1e3846fae883376762a0c1b472baa35b145b911683b7b",
                "sha256:97fa8f1dceffab782069b291e38c4c2227f255cdac5f1e3346666931df87373e",
                "sha256:9d69967673ab7b028c2df09cae05ba56bf4e39e3cb04ebe452b6035c3b49848e",
                "sha256:9e1f53afae865cc32459ad211493cf9e2a3651a7295b7a38654ef3d123808996",
                "sha256:a4a433b3a264dbc9aa9c7c241e87c0358a503ea6394f8737df1683c7c9a102ac",
                "sha256:baadc5f770917ada556afb7651a68176559f4dca5f4b2d0947cd15b9fb84fb51",
                "sha256:c725d11990a9243e6ceffe0ab25a07c46c1cc2c5dc55e305717b5afe856c9608",
                "sha256:d696a8c87315a83983fc59dd27efe034292b9e8ad667aeae51a68b4be14690d9",
                "sha256:e1864a4e9f93ddb2dc6b62ccc2ec1f8250ff4ac0d3d7a15c8985dd4e1fbd6418"
            ],
            "index": "pypi",
            "version": "==1.14.5"
        },
        "pandas": {
            "hashes": [
                "sha256:211cfdb9f72f26d2ede21c751d27e08fed4434d47fb9bb82ebc8ff753888b8b6",
                "sha256:28fd087514616549a0e3259cd68ac88d7eaed6bd3062017a7f312e27941266bd",
                "sha256:2fb7c63138bd5ead296b18b2cb6abd3a394f7581e5ae052b02b27df8244b03ca",
                "sha256:372435456c349a8d39ff001967b161f6bd29d4c3de145a4cf9b366648defbb1f",
                "sha256:3790a3348ab0f416e58061d21693cb662fbb2f638001b94bf2b2199fedc1b1c2",
                "sha256:437a6e906a6717a9ed2627cf6e7895b63dfaa0172567cbd75a553f55cf78cc17",
                "sha256:50b52af2af2e15f4aeb2fe196da073a8c131fa02e433e105d95ce40016df5690",
                "sha256:720daad75b5d35dd1b446842210c4f3fd447464c9c0884972f3f12b213a9edd1",
                "sha256:b4fb71acbc2709b8f5993cb4b5445d8182864f11c39787e317aae39f21206270",
                "sha256:b704fd73022342cce612996de495a16954311e0c0cf077c1b83d5cf0b9656a60",
                "sha256:cbbecca0c7af6a2160b2d6ba30becc286824a98c61dcc6a41fada664f226424c",
                "sha256:d2a071de755cc8ee7784e1b4c7b9b643d951d35c8adea7d64fe7c57cff9c47a7",
                "sha256:d8154c5c68713a82461aba735832f0b4692be8a45a0a340a303bf90d6f80f36f",
                "sha256:e1b86f7c55467ce1f6c12715f2fd1817f4a909b5c8c39bd4b5d2415ef2b04bd8",
                "sha256:fcc63e8134516e93e16eb4ceac9afaa51f4adc5bf58efddae7cbc562f5b77dd0"
            ],
            "index": "pypi",
            "version": "==0.23.1"
        },
        "python-dateutil": {
            "hashes": [
                "sha256:1adb80e7a782c12e52ef9a8182bebeb73f1d7e24e374397af06fb4956c8dc5c0",
                "sha256:e27001de32f627c22380a688bcc43ce83504a7bc5da472209b4c70f02829f0b8"
            ],
            "version": "==2.7.3"
        },
        "pytz": {
            "hashes": [
                "sha256:65ae0c8101309c45772196b21b74c46b2e5d11b6275c45d251b150d5da334555",
                "sha256:c06425302f2cf668f1bba7a0a03f3c1d34d4ebeef2c72003da308b3947c7f749"
            ],
            "version": "==2018.4"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        }
    },
    "develop": {}
}

[peter-ki]

The pipenv is the latest. To be sure I did run

pip3 install pipenv --upgrade
pipenv clean
pipenv update

But the same issue remains.

Reviewing the diagnostic info I realize the Pipfiles differ. There is the extra line

datacache = {editable = true, file = "file:///app/data-cache"}

in the Pipfile in the docker, which is missing in the Pipfile on the host.

[erinxocon]

when the pipfile and lockfile are in sync let me know if you can rebuild the docker container!

[erinxocon]

also try removing the following line:

"1bc1e69" = {path = "./data-cache", editable = true}

From:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
kafka = "*"
pandas = "*"
numpy = "*"
"1bc1e69" = {path = "./data-cache", editable = true}
datacache = {editable = true, file = "file:///app/data-cache"}

[dev-packages]

[requires]
python_version = "3.6"

I don't think you need them both, the one I'm asking to be removed is an older syntax and we've since changed how we resolve local packages.

[peter-ki]

Well, the line

"1bc1e69" = {path = "./data-cache", editable = true}

got created on the host and the line

datacache = {editable = true, file = "file:///app/data-cache"}

got added upon

RUN set -ex && pipenv install --deploy --system

in the docker. So the Pipfiles are in sync before pipenv install is called in docker. To test, I omitted pipenv install line from the Dockerfile and the image got built. I attached to the container and ran

pipenv install --deploy --system --verbose

interactively, while looking at the Pipfile before and after.

Deleting the first line and keeping the second does not work, because on the host "file:///app/data-cache" does not resolve, so pipenv update fails with the following error:

Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
ne_round
    for dep in self._iter_dependencies(best_match):
  File "/usr/local/Cellar/pipenv/11.10.0/libexec/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 275, in _iter_dependencies
    for dependency in self.repository.get_dependencies(ireq):
  File "/usr/local/Cellar/pipenv/11.10.0/libexec/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 215, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/usr/local/Cellar/pipenv/11.10.0/libexec/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 234, in get_legacy_dependencies
    dist = ireq.get_dist()
  File "/usr/local/Cellar/pipenv/11.10.0/libexec/lib/python3.6/site-packages/pipenv/vendor/pip9/req/req_install.py", line 1069, in get_dist
    egg_info = self.egg_info_path('').rstrip('/')
  File "/usr/local/Cellar/pipenv/11.10.0/libexec/lib/python3.6/site-packages/pipenv/vendor/pip9/req/req_install.py", line 483, in egg_info_path
    filenames = os.listdir(base)
FileNotFoundError: [Errno 2] No such file or directory: '/app/data-cache'

[erinxocon]

Sorry don't know what line of thought I was on. I have thought of something else.

FROM ubuntu:18.04

# -- Install Pipenv:
RUN apt update && apt install python3-pip git -y && pip3 install pipenv

ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8

# -- Install Application into container:
RUN set -ex && mkdir /app

COPY ./services /app
COPY ./data-cache /app/data-cache

WORKDIR /app

# -- Adding Pipfiles
COPY Pipfile Pipfile
COPY Pipfile.lock Pipfile.lock

# -- Install dependencies:
RUN set -ex && pipenv install && pipenv --deploy --system

CMD python3 start.py

Can you try that dockerfile with your original pipfile before i asked you to remove that line? It should relock the pipfile after the install which will allow deploy to work. I won't be able to create a small test for another hour or so, apologies for asking you to test so many things.

[peter-ki]

Cool, that works (after fixing the line which should read

RUN set -ex && pipenv install && pipenv install --deploy --system

, I guess) while

RUN set -ex && pipenv install --deploy --system

fails. Could you give a short explanation?

[peter-ki]

Anyway, thank you very much for resolving the issue!

[erinxocon]

I don't think my solution is the proper way, I'm working on another, but essentially, when the pipfile references the local package, the lockfile becomes different for each platform, your host machine and the docker instance since they are in difference places. pipenv install creates a virtualenv, and relocks the pipfile and then pipenv install --deploy --system uses that new updated lockfile for the install. So we are essentially creating a virtualenv when it's not needed to sync lockfiles after you install the new local dep...I'm looking into a better Dockerfile solution now.

[techalchemy]

I am 100% confused about this, @erinxocon and I have been hashing it out all afternoon. This is definitely not a good thing and we will sort out why it's happening. Sorry for the issue, --deploy should never modify your pipfile!

[techalchemy]

Oh... I think I understand what happened... the new requirements library resolves the names of python packages. It must have a bug in it that does two things

1, It resolves to absolute paths and converts to URIs
2. It ignores existing entries with other names

These are not good.

[techalchemy]

(For the record it was much simpler than that, thankfully)