SSH support in Pkg #16041

davidanthoff · 2016-04-25T18:53:54Z

Julia 0.5 has removed all support for cloning private repos on Windows. In previous versions one could always clone private repos via SSH remote URLs, but support for the git protocol seems entirely removed. Cloning private repos via https also doesn't work on Windows, I get the following error:

ERROR: ccall: could not find function getpass
 [inlined code] from .\c.jl:89
 in #prompt#1(::ASCIIString, ::Bool, ::Any, ::ASCIIString) at .\libgit2\utils.jl:17
 in credentials_callback(::Ptr{Ptr{Void}}, ::Cstring, ::Cstring, ::UInt32, ::Ptr{Void}) at .\libgit2\callbacks.jl:56
 [inlined code] from .\refpointer.jl:32
 in clone(::ASCIIString, ::SubString{UTF8String}, ::Base.LibGit2.CloneOptions) at .\libgit2\repository.jl:189
 in #clone#98(::ASCIIString, ::Bool, ::Ptr{Void}, ::Nullable{Base.LibGit2.AbstractPayload}, ::Any, ::ASCIIString, ::SubString{UTF8String}) at .\libgit2.jl:310
 [inlined code] from .\base.jl:111
 in clone(::ASCIIString, ::SubString{UTF8String}) at .\pkg\entry.jl:195
 in clone(::ASCIIString) at .\pkg\entry.jl:221
 [inlined code] from .\promotion.jl:229
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{ASCIIString}})() at .\pkg\dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{ASCIIString}}, ::ASCIIString) at .\file.jl:58
 in #cd#1(::Array{Any,1}, ::Any, ::Any, ::ASCIIString, ::Vararg{ASCIIString}) at .\pkg\dir.jl:31
 in clone(::ASCIIString) at .\pkg.jl:128
 in eval(::Module, ::Any) at .\boot.jl:236

I think there are potentially two solutions:

Re-enable support for the git transport protocol
Fix the error with https connections on Windows (might be enough to fix Implement getpasswd, fix up github two-factor authentication #8228)

I believe that only having 2) would be very painful, because my understanding is that every Pkg.update would for example query for the username/password combo for private repos, i.e. as far as I know there is no support for credential storage in libgit2, right?

So I think if the git transport is not re-enabled, one would really also need integration with the git credential manager interface, so that https credentials that are stored in a registered git credential manager would be picked up by Pkg.clone and Pkg.update. My understanding is that libgit2 has no support for this, so it would probably have to be implemented manually...

I think this issue here should get the 0.5.0 milestone attached. Not being able to work with private repos is a major regression relative to 0.4 that would affect essentially anyone who has package work in private repos.

The text was updated successfully, but these errors were encountered:

EricForgy · 2016-04-25T22:31:09Z

For what it's worth, I am in the same boat as David. I'm on Windows and have many private repos (close to 50 and counting). It took me some time to set up SSH and point all my remotes to git@github.com so that I could do Pkg.update without entering my password so many times. It was also necessary to set that up for Travis and Appveyor. Now, all my tests are failing on 0.5 because the private repos cannot be cloned. This has stopped me from doing any work on 0.5.

Like David, I would hope this gets a 0.5 label because I am REALLY looking forward to 0.5 :)

Cheers

tkelman · 2016-04-25T23:28:55Z

This particular ccall is not present on windows and evidently obsolete, so #8228 (comment) still stands. We're going to primarily support https going forward, so I'm going to close this as a duplicate of #8228.

davidanthoff · 2016-04-25T23:46:58Z

@tkelman: Once #8228 is fixed, will Pkg.update ask me for a username/password combination for every private package that I have? Or is there going to be some caching of credentials?

tkelman · 2016-04-25T23:52:44Z

Probably. I consider ssh keys far worse in usability than just typing a password, so I'm not too motivated to do anything drastic here. I believe @eschnett had looked at adding libssh to deps/Makefile but couldn't get it working, it's nontrivial to add more libraries to the Julia binary distribution.

aviks · 2016-04-25T23:54:44Z

Not everyone uses github. I have locally hosted git repositories that use ssh authentication. Is that going to be unsupported going forward? That's shame, if true.

davidanthoff · 2016-04-26T00:01:43Z

Can we leave this issue open, please? Clearly requiring username/password entering for every private package on every Pkg.update() seems like a completely unworkable solution for many users, so keeping an issue open where solutions can be sorted out seems like a good idea.

I should add that using SSH keys is working great on julia 0.4 for me, so dropping that support without replacing it with something similar would be a huge regression from my point of view.

tkelman · 2016-04-26T00:08:21Z

The windows-specific part of this issue is a duplicate of #8228.

We're using libgit2 for the package manager now, that's not up for debate. If ssh support by default is something that people want strongly enough to implement then I'm not going to say we shouldn't do it, but I don't consider it release-blocking. There's also the option of not using Pkg to manage private packages (just put them somewhere else on LOAD_PATH) since it generally isn't doing much for you with private packages in the current design.

EricForgy · 2016-04-26T00:27:12Z

Hi, I have close to 50 private repos on GitHub and there is no way I am going to enter my password 50 times every time I Pkg.update. Even 4 times ( #8228 ) is too much. If this is really the future for me with Julia, please let me know ASAP as I will need to switch to a different language and the somer I make that painful call, the better. Thanks everyone.

Keno · 2016-04-26T00:31:27Z

No, nobody wants to have to enter a password 50 times. This'll get fixed.

EricForgy · 2016-04-26T00:34:37Z

Thank you for the reassurance Keno. I was starting to panic (I do that too easily sometimes) :)

davidanthoff · 2016-04-26T00:37:48Z

Great! I think some fix should really come for 0.5, so maybe apply the 0.5 label?

At least for github and bitbucket another option would be to plug into the whole git credential storage framework. That might only work if people have git installed and configured, but as an option that would be nice, especially as it would work with https URLs.

Of course that doesn't work for people with locally hosted repos, so I think it wouldn't be a substitute for ssh support.

tkelman · 2016-04-26T00:44:43Z

We've gone without ssh support on master for 6 months now. I don't think this is release blocking.

davidanthoff · 2016-04-26T00:48:46Z

Keep in mind that the vast majority of users hasn't been on master for the last 6 months. Just because something hasn't shown up as a problem for the core dev group in the last 6 months doesn't mean that it can't be a major problem for a large user group that is only now starting to explore 0.5. This issue would keep me from upgrading to 0.5 until sorted out because it completely would break my daily workflow.

aviks · 2016-04-26T00:48:50Z

We've gone without ssh support on master for 6 months now. I don't think this is release blocking.

Oh, come on. Lots of people who use julia in "production" haven't (and shouldn't have) upgraded to 0.5 yet.

timholy · 2016-04-26T00:52:40Z

I only have a few private repos, and it's already pretty irritating.

blakejohnson · 2016-04-26T02:19:11Z

I also find this annoying. I frequently launch 0.4 just to run Pkg.update().

wildart · 2016-04-26T02:37:52Z

Password caching should not be a problem. Temporary solutions is to use ssh-agent.

tkelman · 2016-04-27T12:47:01Z

@Keno any chance enabling the EMBED_SSH_PATH option libgit2/libgit2#2600 on additional platforms might make this a little simpler?

I don't think it's technically feasible in libgit2 just yet, but something Jake and Art and I had discussed was possibly making ssh support optional in a way that PkgDev or some other non-default package could be in charge of obtaining the binaries for libssh2.

edit: side note, even the libgit2sharp maintainers who are core developers of libgit2 choose not to support ssh in the binaries they distribute: libgit2/libgit2sharp#255 (comment)

tbreloff · 2016-04-27T13:43:28Z

In case it helps... I'm very actively using 0.4, but haven't made the jump to 0.5 yet. I have lots of private repos, and I agree that package maintenance would be impossible for me (I'd just abandon it and manage the repos by hand). My password is 20 characters of gibberish, so typing it in is not an option.

tkelman · 2016-07-26T03:55:28Z

can you try #17586? @Keno what's your status there, how much are you intending to finish before rc1?

Keno · 2016-07-26T03:56:28Z

I suppose we can merge the first three commits and put off the tests for later. I've made some progress on the test case, but it's still flakey.

tkelman · 2016-07-26T06:03:22Z

It's not like we have any tests for it now. While we do really badly need some and we shouldn't let it remain untested for any longer than we absolutely have to, working and untested is a bit better than not working and untested.

wildart · 2016-07-26T15:05:17Z

Mbedtls does not support DSA keys. We probably can tweak mbedtls configuration to support for 4096-bit RSA keys.

blakejohnson · 2016-07-27T02:54:23Z

So, even dropping back to 2048-bit RSA doesn't work for me.

wildart · 2016-07-27T18:05:47Z

@blakejohnson are you on linux?

blakejohnson · 2016-07-27T19:12:37Z

No, Mac OS X 10.11.5.

blakejohnson · 2016-07-28T13:23:20Z

After #17586, Pkg.update() does work. However, I am still getting prompted for a passphrase for a key which ssh-agent knows about. Does that mean the interface with ssh-agent is still not functioning?

tkelman · 2016-07-28T15:40:15Z

how often do you get prompted? I don't know how long we cache the credentials for, or if we have it working in a way where we can let ssh-agent entirely handle the passphrase

Keno · 2016-07-28T16:10:38Z

passphrase should be entered when adding the key to the agent, so we shouldn't need to prompt for it.

Keno · 2016-07-28T20:13:06Z

Indeed ssh-agent authentication appears to be broken again. Will take a look.

`usesshagent` is just a flag to determine whether or not to use the agent, so zeroing it isn't necessary. Even worse, since it only gets a reference to one of the flag strings (stored in the AST), zeroing them globally changes those strings, disabling ssh-agent authentication entirely. Fixes the problem noted in #16041

tkelman · 2016-07-29T02:27:42Z

@blakejohnson have you had a chance to re-test that redirection issue again recently? Did you ever open a separate issue on it?

blakejohnson · 2016-07-29T18:13:38Z

No, I haven't tried that again. Though, given the number of other issues Keno has fixed in recent days, I wonder if I wasn't premature in identifying the temporary cause of success. I'll put that on my queue to test, and if it is still a problem, I'll file an issue.

`usesshagent` is just a flag to determine whether or not to use the agent, so zeroing it isn't necessary. Even worse, since it only gets a reference to one of the flag strings (stored in the AST), zeroing them globally changes those strings, disabling ssh-agent authentication entirely. Fixes the problem noted in JuliaLang#16041

giordano · 2016-09-21T22:06:19Z

I do experience the problem of being prompted for the passphrase of the wrong key (GitHub key is not the default one), while I don't have this issue with other applications using SSH (nor I have with Julia 0.4.6). I'm running Julia Version 0.6.0-dev.629, commit 155147e (2016-09-14 17:49 UTC), should I file a new issue?

iamed2 · 2016-09-21T22:28:09Z

@giordano Do you have that key set in your SSH config file? Unfortunately, libssh2 (and consequently libgit2) cannot use the SSH config file information.

giordano · 2016-09-21T22:32:29Z

Yes, I have. Am I hopeless? At least I have to type the password just once ;-)

iamed2 · 2016-09-21T23:21:37Z

@giordano You and I are both in the same crappy boat, I'm afraid. I expect it won't be fixed unless libssh2 starts parsing SSH config files.

EricForgy · 2016-09-21T23:43:31Z

Although inputting a password once is not the end of the world, it is not completely symmetric if linux/OSX do not need to input passwords. Until this is fixed for Windows, I almost say we should break it for linux/OSX too, just to be fair and so it doesn't drop off the radar (given this issue is closed - presumably in a rush so they could release 0.5 - although it doesn't seem 100% satisfactory).

EricForgy · 2016-09-21T23:44:48Z

Windows should not be a second class Julia citizen.

tkelman · 2016-09-21T23:44:59Z

What leads you to think this is Windows-specific?

EricForgy · 2016-09-22T00:12:20Z

Ignorant assumptions? :)

tkelman · 2016-09-22T01:23:11Z

windows is kind of a second class citizen with ssh usability in general apart from julia. we try not to intentionally break things on any platform, but if there are remaining improvements to make on any platform we should have issues to track them (see #17541, and there are a few already open and being tracked)

iamed2 · 2016-09-22T02:00:54Z

@EricForgy I'm on OS X. I actually don't have to input a passphrase as my key doesn't have one, but I have to type the full path to my key every time.

IssamT · 2016-10-05T14:23:45Z

On OSX el capitan 10.11.6 I am unable to get Pkg.clone() working on julia 0.5

http://stackoverflow.com/questions/39849236/pkg-clone-changed-on-julia-0-5

BoundaryValueProblems · 2017-03-10T18:04:09Z

With julia 0.5.1, Pkg.clone for private repos works on Mac OS X (El Capitan 10.11.6) and Ubuntu/Linux (16.04 LTS), but on Windows 10, the situation is quite peculiar: it works with https while it fails using ssh with the following errors:

ERROR: ArgumentError: drive mismatch: METADATA git@gitlab.com:...
 in joinpath(::String, ::String) at .\path.jl:82
 in clone(::String) at .\pkg\entry.jl:210
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{String}})() at .\pkg\dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{String}}, ::String) at .\file.jl:48
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at .\pkg\dir.jl:31
 in clone(::String) at .\pkg\pkg.jl:151

Hence, this issue should be reopened!

yuyichao · 2017-03-10T18:18:20Z

No, that's the windows path issue.

StefanKarpinski · 2017-03-10T20:21:48Z

Unrelated problem, addressed by this PR: #20914.

ViralBShah added packages Package management and loading regression Regression in behavior compared to a previous version labels Apr 25, 2016

tkelman closed this as completed Apr 25, 2016

tkelman reopened this Apr 26, 2016

tkelman changed the title ~~Cloning or updating private packages doesn't work on Windows~~ SSH support in Pkg Apr 26, 2016

Keno added this to the 0.5.0 milestone Apr 26, 2016

tkelman modified the milestones: 0.5.x, 0.5.0 Apr 26, 2016

Keno self-assigned this Apr 26, 2016

Keno modified the milestones: 0.5.0, 0.5.x Apr 26, 2016

Keno mentioned this issue Jul 28, 2016

Don't zero usesshagent #17682

Merged

SSH support in Pkg #16041

SSH support in Pkg #16041

Comments

davidanthoff commented Apr 25, 2016

EricForgy commented Apr 25, 2016

tkelman commented Apr 25, 2016

davidanthoff commented Apr 25, 2016

tkelman commented Apr 25, 2016

aviks commented Apr 25, 2016

davidanthoff commented Apr 26, 2016

tkelman commented Apr 26, 2016

EricForgy commented Apr 26, 2016

Keno commented Apr 26, 2016

EricForgy commented Apr 26, 2016

davidanthoff commented Apr 26, 2016

tkelman commented Apr 26, 2016

davidanthoff commented Apr 26, 2016

aviks commented Apr 26, 2016

timholy commented Apr 26, 2016

blakejohnson commented Apr 26, 2016

wildart commented Apr 26, 2016

tkelman commented Apr 27, 2016 • edited Loading

tbreloff commented Apr 27, 2016

tkelman commented Jul 26, 2016

Keno commented Jul 26, 2016

tkelman commented Jul 26, 2016

wildart commented Jul 26, 2016 • edited Loading

blakejohnson commented Jul 27, 2016

wildart commented Jul 27, 2016

blakejohnson commented Jul 27, 2016

blakejohnson commented Jul 28, 2016

tkelman commented Jul 28, 2016

Keno commented Jul 28, 2016

Keno commented Jul 28, 2016

tkelman commented Jul 29, 2016

blakejohnson commented Jul 29, 2016

giordano commented Sep 21, 2016

iamed2 commented Sep 21, 2016

giordano commented Sep 21, 2016

iamed2 commented Sep 21, 2016

EricForgy commented Sep 21, 2016

EricForgy commented Sep 21, 2016

tkelman commented Sep 21, 2016

EricForgy commented Sep 22, 2016

tkelman commented Sep 22, 2016 • edited Loading

iamed2 commented Sep 22, 2016

IssamT commented Oct 5, 2016

BoundaryValueProblems commented Mar 10, 2017

yuyichao commented Mar 10, 2017

StefanKarpinski commented Mar 10, 2017

tkelman commented Apr 27, 2016 •

edited

Loading

wildart commented Jul 26, 2016 •

edited

Loading

tkelman commented Sep 22, 2016 •

edited

Loading