Editable installs are broken

[hoefling]

0162486 introduced changes in metadata of the editable installs that are supported by neither importlib.metadata nor pkg_resources; in particular, the errors are due to the new layout that dropped egg-link. Example tests:

import os
import shutil
import site
import sys
from pathlib import Path

import pytest

from poetry.factory import Factory
from poetry.io.null_io import NullIO
from poetry.masonry.builders.editable import EditableBuilder
from poetry.utils.env import EnvManager
from poetry.utils.env import VirtualEnv


@pytest.fixture
def tmp_venv(request, tmp_path):
    project = Factory().create_poetry(Path(request.config.rootdir) / "tests" / "fixtures" / "simple_project")
    env_manager = EnvManager(project)
    venv_path = tmp_path / "venv"
    env_manager.build_venv(venv_path)
    venv = VirtualEnv(venv_path)
    builder = EditableBuilder(project, venv, NullIO())
    builder.build()
    site.addsitedir(str(venv.site_packages.absolute()))
    yield venv
    shutil.rmtree(venv.path)


def test_importlib_metadata(tmp_venv):
    import importlib.metadata
    assert Path("simple_project", "__init__.py") in importlib.metadata.files("simple-project"), f'Found in dist: {os.linesep.join(str(p) for p in importlib.metadata.files("simple-project"))}'


def test_pkg_resources(tmp_venv):
    import pkg_resources
    dist = pkg_resources.get_distribution("simple-project")
    assert "simple_project" in dist.get_metadata("top_level.txt").split()

Both test_importlib_metadata and test_pkg_resources fail on current master, but pass on 1.0.10 tag.

I also can't find any spec to the new editable install layout (partial dist-info plus the pth), only some discuss threads on a similar idea that isn't approved yet:

• Specification of editable installation
• Third try on editable installs
• Next steps for editable / develop proof of concept

[abn]

@hoefling to save me from repeating myself this comment applies: #2725 (comment)

Essentially, "editable installs" are non-standard practice introduced by setuptools. This was hacky at best and unreliable.

Regarding the importlib.metadata issue, my best guess is that the issue there is the activation of the environment, particularly how the site injection is done. The import logic in python is .... non-trivial. :)

Related: #2452

[hoefling]

@abn no, importlib.metadata looks either for a .dist-info/RECORD or a .egg-info/SOURCES.txt as a fallback. The "old" (current) editable layout points to egg-info which contains the complete file list in a SOURCES.txt. The "new" layout has a dist-info which contains a RECORD with an file list that doesn't list any sources. Again, the new impl breaks the standard library module; I don't care much about pip or setuptools, just listing that as an additional example since pkg_resources is still used. We rely on importlib.metadata to gather info about installed plugins for our library, for now I will have to include a warning in our readme about blacklisting poetry==1.1 once it is released.

[abn]

@hoefling as I mentioned before editable installs are a hack. The current mechanism relies on the .pth file as described here.

Editable installs, as far as I can tell, were not a consideration for importlib.metadata implementation. And, the generation of an .egg-info was a side-effect of poetry previously using setuptools for editable installations, which is something we planned on removing. Until the community has a standard around this, I suspect this is not going to change.

Including files being actively developed as RECORD entries seems rather pointless since these are, well editable, and more files can be added or removed without running poetry install. This inevitably will lead to things being out of sync, and any reliance on such mechanism for an editable install will be brittle at best. If we get a standard around editable installs, that might help things, but I do doubt that you can reliably identify all files in a package relying solely on importlib.metadata.files. Also, I am not sure what the side-effects of writing stale entries to the RECORD file is going to be in an editable scenario.

EDIT: An unwanted side-effect here would be pip uninstall or a similar command, even when adding a dependency with poetry using develop = true will end up removing your files, which is definitely not something you want in your editable scenario.

For your use case, I believe you will have to detect the existence of a .pth file and handle the entries here as described by the site documentation.

(.venv) $ python -c "import importlib.metadata; print(importlib.metadata.files('foobar'))"
[PackagePath('/tmp/foobar/.venv/lib/python3.8/site-packages/foobar.pth'), PackagePath('/tmp/foobar/.venv/lib/python3.8/site-packages/foobar-0.1.0.dist-info/METADATA'), PackagePath('/tmp/foobar/.venv/lib/python3.8/site-packages/foobar-0.1.0.dist-info/INSTALLER'), PackagePath('/tmp/foobar/.venv/lib/python3.8/site-packages/foobar-0.1.0.dist-info/RECORD')]

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.