It is possible to add a dependency of a private git repo? #835
Comments
|
You need to specify the protocol: Or just add using the --git flag See here for more info on the add command https://poetry.eustace.io/docs/cli/#add |
|
@Hammond95 Is your issue fixed now? |
|
@jasongi-actu 's solution did not fix the issue for me. I get this error: Dependency is declared as: UpdateUsing Poetry version |
|
@jasongi-actu Did you try that command you suggested? Did it work for you? I even tried on a public repository I have in GitHub (only specifying the |
|
Yes I use that command all the time in multiple public/private github repos. I think the next step in solving this would be to get a reproducible test case. Also, are you running PIP <= 18.1? There are issues with git dependencies and poetry using pip 19.xx |
|
@jasongi-actu I am indeed running $ poetry run pip --version
pip 18.1 from [...]I can reproduce the error by trying to install a public repo of my own: [tool.poetry.dependencies]
markdownreveal = { git = "ssh://git@github.com/Peque/markdownreveal.git", branch = "master" }Whereas I am able to: git clone ssh://git@github.com/Peque/markdownreveal.git`I guess if you fork it and are able to install from your own fork using the SSH protocol, then it is related to differences between our systems/configurations/environments, not depending on the project that it is to be installed. In which case I am not sure how to proceed to debug. |
|
@jasongi-actu Can you reproduce the issue when forking [tool.poetry.dependencies]
markdownreveal = { git = "ssh://git@github.com/jasongi-actu/markdownreveal.git", branch = "master" }Alternatively, could you provide a repo that works fine for you so that I can fork it and try it myself? |
|
@Peque I ran hope it helps! |
|
Sorry for the late reply, tried @jasongi-actu proposed solution (adding the ssh protocol), but this still doesn't work... This is the error I get:
Running manually the corresponding command
results into the following error:
running manually the same command but without the
works fine. I get the same errors with the PS. I am using poetry version |
|
Updating poetry to I didn't created this env (maybe poetry did?); I am using a conda env to make poetry use a python3.6 interpreter. |
|
ummmm the you can just put it directly into toml file and run |
|
@Hammond95 I will close this issue now since your particular issue seems to be resolved and @dsevero has also offered an alternative solution. Furthermore, you can add @thedrow that looks like a different bug and the |
|
Can I request a reopen of this issue? I'm using error. Digging into the code it looks like lines I added a simple print out of the result dict and get In my case I'm matching on a tag which is actually how we're versioning our dependency. In our setup.py file we have a more or less static version set, but if there's a method that poetry would accept then I'd happily add that. I've attempted using the branch directive as well and get the same behavior. Thanks. |
|
Hey @darakian , please open a new issue, where you describe your problem by showing your Thanks! fin swimmer |
|
Hey @finswimmer I ended up realizing that it was an error on my end (sorry). Turns out that our code was using distutils rather than setuptools and the lines poetry/poetry/puzzle/provider.py Lines 324 to 332 in 636ce8b were borking out as the egg_info command was failing. A one line change of
to in our |
|
poetry 1.x Not enough information in the help too: 😞 |
|
@dazza-codes: The correct syntax would be: fin swimmer |
|
can use token as well $ poetry add git+https://GIT_TOKEN@github.com/org/privaterepo.git#BRANCH |
but this might make the whole dependency chain token dependent, if somebody revokes that token, we have to manually change this token, does ssh link sounds better? |
@InnerTaste what is GIT_TOKEN in this example? An environment variable? Or a value which gets hardcoded into pyproject.toml? |
@venaturum
yeah may have drawbacks too, but this https/token method works in docker |
Thank you, this really did the trick. |
Do you know how to specify [tool.poetry.dependencies]
python = "^3.8"
package_name = {git = "https://<token>@github.com/<user>/<repo>", rev = "0.1.3"}But I don't want to specify the token directly in |
|
I would not recommend using a hard coded TOKEN unless it can be shared among users of an organization or protected in CI/CD workflows. Using an ssh protocol allows any user or CI/CD workflow with the required ssh credentials to get the library OK. It works well when the ssh credentials are coupled with an ssh-agent. |
So, is this mean that it is not possible to specify the TOKEN using an environment variable, and the only way I can use it is to hard code it into |
|
It means that if you are fortunate enough to work on a *nix system you can use ssh access by key and ads the key to your ash agent after you set it up so it should login automatically. However this is not possible using http and token based authentication. In our Organisation we actually solved the problem by setting up a private pypi repository where a package is updated whenever we push a tag to the git repository. We were then able to add the private repository to the pip config as "extra index". |
|
I also throw in my two cents. Most of our package come from PyPI, but a few need to be pulled from a private and SSH key-file-only Gerrit repository (no anonymous, no username/password). po add git+ssh://code.some-domain.de:29418/foo/bar#v2.1.3+poetry -vv
Using virtualenv: /home/cmdq/.cache/pypoetry/virtualenvs/foobar-j0Abac2_-py3.8
Stack trace:
11 ~/.poetry/lib/poetry/_vendor/py3.8/clikit/console_application.py:131 in run
status_code = command.handle(parsed_args, io)
10 ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:120 in handle
status_code = self._do_handle(args, io)
9 ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:171 in _do_handle
return getattr(handler, handler_method)(args, io, self)
8 ~/.poetry/lib/poetry/_vendor/py3.8/cleo/commands/command.py:92 in wrap_handle
return self.handle()
7 ~/.poetry/lib/poetry/console/commands/add.py:106 in handle
requirements = self._determine_requirements(
6 ~/.poetry/lib/poetry/console/commands/init.py:320 in _determine_requirements
requires = self._parse_requirements(requires)
5 ~/.poetry/lib/poetry/console/commands/init.py:410 in _parse_requirements
package = Provider.get_package_from_vcs(
4 ~/.poetry/lib/poetry/puzzle/provider.py:193 in get_package_from_vcs
git.clone(url, tmp_dir)
3 ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/vcs/git.py:212 in clone
return self.run("clone", "--recurse-submodules", repository, str(dest))
2 ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/vcs/git.py:302 in run
subprocess.check_output(["git"] + list(args), stderr=subprocess.STDOUT)
1 ~/.pyenv/versions/3.8.6/lib/python3.8/subprocess.py:411 in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
CalledProcessError
Command '['git', 'clone', '--recurse-submodules', 'code.some-domain.de:29418/foo/bar', '/tmp/pypoetry-git-foobarnplduj04']' returned non-zero exit status 128.
at ~/.poetry/lib/poetry/utils/_compat.py:217 in run
213│ process.wait()
214│ raise
215│ retcode = process.poll()
216│ if check and retcode:
→ 217│ raise CalledProcessError(
218│ retcode, process.args, output=stdout, stderr=stderr
219│ )
220│ finally:
221│ # None because our context manager __exit__ does not use them.```A simple checkout on the console is as easy as AnalysisIt's no wonder this doesn't work. Looking at the built command line we see
What's missing is at least the correct user (which I would not want hard-coded in the pyproject.toml BTW) and any authentication. Already tried
We cannot upload that proprietary to PyPI and won't be getting an internal package repository soon. Also any other authentication method is out. The later is the core of the problem in our setup IMHO. Did I overlook a way to set this up? Otherwise please reopen. |
|
@CmdQ, at my work we publish packages to a private pypi feed in Azure DevOps. Prior to this though we would just publish to a simple pypi server (https://pypi.org/project/pypiserver/) that was running on an on-site VM connected to our network. Both of these work with poetry. Not a solution to the issue at hand but perhaps it's an option. |
Update for devs/affectedRoot causeWe found the core problem: the port specification. If we in any way try to inject the port into the URL, the process fails. That's because for Git—without knowing that SSH is meant—doesn't treat the number as a port but part of the repo path, and that fails. If you on the other hand manually run Why isn't the SSH protocol specification forwarded to Git? I'd consider that a bug of Poetry. SolutionDon't try to add the port via Poetry/command line but set it in your You can/must do this for your CI image, too. |
|
Hi, I think it would still be a valuable feature if one could use env vars or the poetry config methods to set the token for https+git. E.g. it should be simpler to pass a token into Docker and CI/CD than setting up ssh. |
|
I'm trying to add a private repo via But I get an error similar to one posted above: Should I raise a new issue for this? I'm using poetry |
|
Guys, I have just spent a tremendous time trying to make this work and my issue was the passphrase for ssh. I am on windows, I solved using the knowledge to be found on the following pieces of wisdom: |
I have the same issue; it appears that the /tmp/ directory does not exist |
|
Please raise a new issue with specific details if this is still affecting users. Note that #5428 should improve git dependency handling considerably. Please try that branch before raising new issues. |
|
Works fine now with 1.1.13! poetry add git+ssh://git@github.com/privateorg/privaterepo.git#main |
|
By the way poetry defaults to checking out Easy fix is to specify the rev to point to |
|
A simple way is doing the following: on your if your package name has spaces: if you need a particular revision: if you don't want to specify the token for authentication you can use environment variables: |
|
Did anyone try installing dependencies that have recursive dependencies provided as Git repos? When I try to install project in repo C using the command
Is there a way to resolve it? P.S. both the parent repos requires private token for access. |
|
@mohitjain2504 yes it will work, but you have to stop using requirements.txt and instead define your dependencies in a Note that those projects don't need to adopt poetry. They just need to specify their dependencies like this: The main difference you'll see about this setup is that your project originally using Rather than using a "vanilla" |
|
What if we try to clone a private repository from Bitbucket (or GitHub) but using environment variables instead of directly embedding the credentials using a .env file? Is this approach considered a best practice? for example:
|
We achieve our private git dependency reference using ssh url. IMHO it is more flexible to set up, and more uniform, should you have some other public git dependencies, or should you decide later to relax protection and open up your repo. IMO credentials laying around in files pose a harder-to-justify security risk, because you (your CI) depend on cleaning up after yourself (your build). We use ssh git urls successfully across windows, wsl, docker, mac environments with acceptable level of inconvenience (mandating passphrase protected ssh keys + ssh agent). |
|
I agree, but I think we should have both options on the table. I have tried ssh already, but because of some reasons, using env variables would make more sense in my case and it helps to be more flexible to some extent! BTW, thank you for clarification. I appreciate your time. |
While using ssh and the required key pair works for me as well, it requires to be set up for each user/machine individually, correct? IIRC, my key pair is set up for my personal github acount which in turn needs to be granted access to the dependency repository. |
|
Leaving this here in case it is useful for others. I got it working running this command: $ poetry add git+ssh://git@github.com/username/private_repo.git@masterFor me it worked after adding $ GH_TOKEN="TOKEN" echo "https://${GH_TOKEN}:@github.com" > ${HOME}/.git-credentials && git config --global credential.helper storeI am using Fine-grained Tokens: https://github.com/settings/tokens?type=beta |
|
Hey peeps, I am not a Python dev, I am trying to figure out how to build a docker image using poetry with a private repo, hopefully using Docker buildkit and the Do you have any implementation reference I could follow? |
|
If you want to set up SSH on your server, go ahead and do that. Then, you can use SSH to clone your repository. If you're using Poetry, just update the placeholders in the example below with your actual repository details, and you're good to go! my_private_repo = { git = "https://$USERNAME:$PASSWORD@github.com/companyrepo/myproject.git", branch = "main" }Remember to replace $USERNAME and $PASSWORD with your real GitHub username and password. |
|
@yordis specify the git dependency like this in [tool.poetry.dependencies]
something = { git = "ssh://git@github.com/you/something.git", rev = "main" }Make sure you have ssh access to the git repo locally. Set up Docker with access to your local SSH keys during the build stage, like explained here. I guess then you want to run something like this from your Dockerfile: RUN --mount=type=ssh poetry install --only main |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
and others
Question
It is possible to add a dependency of a private git repo?
I've just stated to use poetry, I quite like some of the features, but i'm now stuck since installing a private repo doesn't seem to work...
toml file extract
I've also tried with different git url forms:
the corresponding pip command i have used till now is:
pip install git+ssh://git@github.com/myorganization/myprivaterepo.gitThis is the error i get:
The text was updated successfully, but these errors were encountered: