Make repository pip installable via URL

[erjel]

Hi,

I definitely see the benefits for a single webknossos-libs repository for your development workflow. Yet, this blocks ad-hoc bug fixes from my side in private forks for the sake of quick feedback cycles.

A little background: I use snakemake to clue together my analysis pipelines and to keep track of my research pipeline. It offers the option to define custom conda environments for each tool to create a reproducible workflow. If a production run of mine crashes due to a bug in any of the used python libraries, I tend to create a private fork first to check whether it is really a bug in the library or I screwed up something else.

In most cases the private forks are github repositories which can be quickly installed by changing the pip name with the corresponding repository URL and branch/ commit/ tag identifier.

Unfortunately the subdirectory pip URL fragement does not work with webknossos-libs because the cluster_tools and webknossos are given by relative paths in wkcuber/pyproject.toml.
If I run

pip install "git+https://github.com/scalableminds/webknossos-libs.git#egg=wkcuber&subdirectory=wkcuber"

I get the error message:

pip._vendor.packaging.requirements.InvalidRequirement: Invalid URL: ../cluster_tools

Currently I have only two options to quickly iterate on private webknossos-libs extensions:
1.) Maintain my own versions of wkcuber, cluster_tools, webknossos (python lib) each in separate private fork
2.) Write you an issue and hope that my pull request gets quickly incorporated in an official pypi release

While I generally favor option 2, I am afraid that I have to switch to option 1 - together with burden of keeping up with your relatively fast release cycles ...

That said, there are two other options:
3.) Make the push to separate repositories part of your release cycle: You keep the benefit of a 'mono-repo'-like development process, and everyone can fork the individual repositories for quick ad-hoc solutions
4.) Replace the relative paths in the pyproject.toml by something which directly works with pip install

Eric

[hotzenklotz]

You can pip install local clones of the repository. IIRC, you can pass the -e argument to install a packages as
editable and link your local dev versions for global use in Python. (Pip Docs) Maybe that could help?

e.g. pip install -e <local/repo/directory/>/webknossos-libs

[erjel]

Hi,

thanks for the quick reply!

I'm affraid, the command does not work but rather results in the error message:

ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode

Only cluster_tools can be installed directly via pip:

cd cluster_tools
pip install .

The subdirectories webknossos and wkcuber both fail (even in non-editable) due to the relative path dependency for cluster_tools:

cd wkcuber
pip install .

Error message:

ERROR: Command errored out with exit status 1:
   command: $HOME/usr/miniconda3/envs/webknossos-libs-env/bin/python $HOME/usr/miniconda3/envs/webknossos-libs-env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpnitdntu5
       cwd: /tmp/pip-req-build-sq67wday
  Complete output (16 lines):
  Traceback (most recent call last):
    File "$HOME/usr/miniconda3/envs/webknossos-libs-env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
      main()
    File "$HOMEusr/miniconda3/envs/webknossos-libs-env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "$HOME/usr/miniconda3/envs/webknossos-libs-env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 151, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
    File "/tmp/pip-build-env-r9x9ya77/overlay/lib/python3.9/site-packages/poetry/core/masonry/api.py", line 43, in prepare_metadata_for_build_wheel
      poetry = Factory().create_poetry(Path(".").resolve(), with_dev=False)
    File "/tmp/pip-build-env-r9x9ya77/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 93, in create_poetry
      self.create_dependency(name, constraint, root_dir=package.root_dir)
    File "/tmp/pip-build-env-r9x9ya77/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 244, in create_dependency
      dependency = DirectoryDependency(
    File "/tmp/pip-build-env-r9x9ya77/overlay/lib/python3.9/site-packages/poetry/core/packages/directory_dependency.py", line 41, in __init__
      raise ValueError("Directory {} does not exist".format(self._path))
  ValueError: Directory ../cluster_tools does not exist
  ----------------------------------------

But this is not the issue I am concerned about: I would like to install wkcuber and webknossos directly from a repo URL via pip. For the local repository case I have workarounds:

How to install local webknossos-libs with pip

To install the packages manually I need to replace the relative paths in webknossos/pyproject.toml and wkcuber/pyproject.toml with "0.0.0" and install them in the order cluster_tools, webknossos, and wkcuber.

How to install local webknossos-libs in pip editable mode

To install the packages in editable mode I need the mentioned modifications in pyproject.toml and

cd cluster_tools
poetry build
tar -xvf dist/*-`poetry version -s`.tar.gz -O '*/setup.py' > setup.py
mv pyproject.toml pyproject.toml_bk
cd ../webknossos/
poetry build
tar -xvf dist/*-`poetry version -s`.tar.gz -O '*/setup.py' > setup.py
mv pyproject.toml pyproject.toml_bk
cd ../wkcuber/
poetry build
tar -xvf dist/*-`poetry version -s`.tar.gz -O '*/setup.py' > setup.py
mv pyproject.toml pyproject.toml_bk
pip install -e .

So it is more decision than a technical problem. I would love to see separate repositories for cluster_tools, wkcuber, and webknossos by scalable minds which can directly installed with pip by giving the github URL. If you decide not to support this niche use case, I am also fine with that and create private workarounds.

[erjel]

just for your info:
This is how I solved this (for now) with gitlab actions:
https://github.com/research-center-caesar/webknossos/blob/main/.github/workflows/copy_repo.yml

It is not perfect (I am by no means proficient with github actions), i.e.:

• replace the cp with rsync -r --delete and exclude .gitlab and .hash. Currently, it is only additive ...
• limit the update to version tags instead of commit hashes since you do not want to replicate every single development step in the main repository ...

[jstriebel]

Glad you found a workaround, @erjel. I just looked into this a bit deeper, the problem is the parsing of the requirements in pip already. Changing our structure in the repository would break our development workflow, as we use editable installs to propagate changes from a module directly into another, so we're not able to do solution 4.) you proposed. There's one more possibility, which might be easier than having your own mirror of the repository, which is patching pip itself to work around the error:

First you need to patch pip's pip/_vendor/pkg_resources/__init__.py parse_requirements function with a try-except in the end:

def parse_requirements(strs):
    """Yield ``Requirement`` objects for each specification in `strs`

    `strs` must be a string, or a (possibly-nested) iterable thereof.
    """
    # create a steppable iterator, so we can handle \-continuations
    lines = iter(yield_lines(strs))

    for line in lines:
        # Drop comments -- a hash without a space may be in a URL.
        if ' #' in line:
            line = line[:line.find(' #')]
        # If there is a line continuation, drop it, and append the next line.
        if line.endswith('\\'):
            line = line[:-2].strip()
            try:
                line += next(lines)
            except StopIteration:
                return
        try:
            yield Requirement(line)
        except Exception as e:
            logger = logging.getLogger(__name__)
            logger.warning(e)

Then you can install all modules from the repo like this:

pip install "git+https://github.com/scalableminds/webknossos-libs.git#egg=cluster_tools&subdirectory=cluster_tools"
pip install "git+https://github.com/scalableminds/webknossos-libs.git#egg=webknossos&subdirectory=webknossos"
pip install "git+https://github.com/scalableminds/webknossos-libs.git#egg=wkcuber&subdirectory=wkcuber"

This simply ignores the dependencies from the repository it cannot parse. This issue in pip seems to be related: pypa/pip#6658

I'm going to close this issue for now since it seems you found a workaround for yourself and the patching should also work, but feel free to re-open if this does not solve it for you.