Implement getpasswd, fix up github two-factor authentication

[kmsquire]

#5252 #6668 added support for github two-factor authentication. However, the implementation is a bit ugly: in the worst case, the user has to enter her password 4 times(!).

The nicest way to handle this would probably be to implement a getpasswd (or getpass) function, which reads in a password without echoing, and then passes that onto curl via curl -K - (which reads a config file from stdin).

[stevengj]

The Python getpass function may be helpful. There is also getpass.c from glibc, and a thread on stackoverflow.

(We should certainly be able to put together a pure-Julia implementation, but it would be good to take a quick look at other implementations in case there are subtleties.)

[stevengj]

Note that we shouldn't keep the password in memory longer than necessary; probably better to be paranoid and to overwrite the string buffer with zeros when we are done; one also needs to be a bit careful in the getpass implementation to avoid making extra copies of the data.

[StefanKarpinski]

@Keno, what's the best way to do prompting with the REPL? I'm assuming we don't want a new REPL mode for this.

[Keno]

Well, we should implement a simple "prompt" function, but basically what you do is construct a Prompt object and then pass it to prompt! with the currently active terminal.

[StefanKarpinski]

Hmm. I'm messing with this and my need to refactor is getting strong...

[quinnj]

How does this differ from readline(STDIN)? Or is this on the repl side of things, I guess? In any case, does this make it easier to read from the prompt in IJulia?

[StefanKarpinski]

It's quite possible that STDIN is coming from something other than the terminal. Ideally, this should be nicely integrated with our line editing and allow backspace to work, etc.

[stevengj]

readline(STDIN) works in IJulia, but that is about it; IPython is fairly limited in how much we can emulate standard input.

However, password input requires a bit different interface, because you don't want the password to appear as it is typed. And sending passwords via IJulia requires even more thought about security; I wouldn't recommend it for now.

[tkelman]

I think this should be reopened at PkgDev.jl. @wildart @jakebolewski how much of this is already functioning?

[stevengj]

A getpasswd function might be more generally useful than just PkgDev.

[wildart]

I use libc getpass in libgit2/utils.jl.

[stevengj]

The man page for getpass says "This function is obsolete. Do not use it." It also doesn't exist on Windows as far as I can tell. Python uses their own implementation, apparently.

[tkelman]

Does newer libuv maybe have something for this?

[wildart]

I think it is easier to write pure julia implementation.

[tkelman]

It also doesn't exist on Windows as far as I can tell.

It does not. @wildart please be more careful to not assume that a posix libc is present. Especially in interactive code that does not get tested on CI.

[davidanthoff]

Can this please get a 0.5 label? Until this is fixed, there is a major regression for windows users, i.e. that they can't work with private package repos at all.

[Keno]

Somebody with access to a windows machine please test the following:

diff --git a/base/libgit2/utils.jl b/base/libgit2/utils.jl
index b0c841b..66a9ab7 100644
--- a/base/libgit2/utils.jl
+++ b/base/libgit2/utils.jl
@@ -11,13 +11,20 @@ end

 isset(val::Integer, flag::Integer) = (val & flag == flag)

+using Base.Terminals: TTYTerminal, raw!
 function prompt(msg::AbstractString; default::AbstractString="", password::Bool=false)
-    msg = !isempty(default) ? msg*" [$default]:" : msg*":"
-    uinput = if password
-        bytestring(ccall(:getpass, Cstring, (Cstring,), msg))
+    msg = !isempty(default) ? msg*" [$default]: " : msg*": "
+    print(msg)
+    if password
+        t = TTYTerminal("xterm", STDIN, STDOUT, STDERR)
+        raw!(t, true)
+        try
+            uinput = rstrip(readuntil(STDIN,'\r'),'\r')
+        finally
+            raw!(t, false)
+        end
     else
-        print(msg)
-        chomp(readline(STDIN))
+        uinput = chomp(readline(STDIN))
     end
     isempty(uinput) ? default : uinput
 end

Tests for this are nontrivial, since it depends on the exact state of the tty which we can't really emulate.

[tkelman]

Don't we want this not to echo what gets typed?

[Keno]

Yes, that's what happens in raw mode.

[Keno]

At least on unix.

[tkelman]

Raw mode isn't really working on Windows then. And one or both of the \r above would have to be a \n to work in mintty.

[stevengj]

Can we just use _getch on Windows?

The more I look into this, the more I think we should just use getpass on Unix systems. It's been marked as "obsolete" for over a decade, but since there is no simple alternative function it is still widely used. If it ever disappears we can re-implement it, but why bother now?

[stevengj]

The following works for me on Mac, GNU/Linux, and Windows, though it would probably be good to handle the backspace key and other niceties on Windows:

@windows_only function getpass(prompt::AbstractString)
    print(prompt)
    p = UInt8[]
    sizehint!(p, 128)
    while true
        c = ccall(:_getch, Cint, ())
        c < 16 && break
        push!(p, c)
    end
    s = bytestring(p)
    fill!(p, 0)
    return s
end
@unix_only getpass(prompt::AbstractString) = pointer_to_string(ccall(:getpass, Cstring, (Cstring,), prompt), true)

[tkelman]

That sounds like it'll be good enough for now. Certainly better than trying to ccall something that doesn't exist.

[stevengj]

Here's a somewhat improved version, that handles backspace but ignores other control characters as well as arrow and function keys:

@windows_only function getpass(prompt::AbstractString)
    print(prompt)
    flush(STDOUT)
    p = Array(UInt8, 128) # mimic Unix getpass in ignoring more than 128-char passwords
                          # (also avoids any potential memory copies arising from push!)
    plen = 0
    while true
        c = ccall(:_getch, UInt8, ())
        if c == 0xff || c == UInt8('\n') || c == UInt8('\r')
            break # EOF or return
        elseif c == 0x00 || c == 0xe0
            ccall(:_getch, UInt8, ()) # ignore function/arrow keys
        elseif c == UInt8('\b') && plen > 0
            plen -= 1 # delete last character on backspace
        elseif !iscntrl(Char(c)) && plen < 128
            p[plen += 1] = c
        end
    end
    s = bytestring(pointer(p), plen)
    fill!(p, 0) # don't leave password in memory
    return s
end

@unix_only getpass(prompt::AbstractString) = pointer_to_string(ccall(:getpass, Cstring, (Cstring,), prompt), true)

There is also a _getwch function that returns a wint_t, but it appears to be useless: I tried it out, and not only does it seem to ignore Unicode characters, but it mimics _getch in using 0xe0 to indicate an arrow key (making the latter indistinguishable from U+00e0). In consequence, I think we can only support ASCII passwords on Windows.

[vtjnash]

Raw mode isn't really working on Windows then

i find that hard to believe since we depend on it for the REPL to print correctly

[tkelman]

Ah, apologies, it looks like Keno's code does work when built into the sysimg but not when called from the repl. My bad.

[wildart]

I wonder if xterm is installed on every platform or distro by default.

[nalimilan]

I wonder if xterm is installed on every platform or distro by default.

Just tried it, it's not even installed on my Fedora 23 box (on which I installed all kinds of packages over the years). So no.

[stevengj]

Note that one of the things that we have to be very careful of is accidentally leaving extra copies of the password in memory. e.g. @Keno's version calls readuntil (which calls jl_readuntil), and it looks like jl_readuntil can easily make extra copies.

getpass implementations presumably go to more care in this regard. I tried to be careful in my Windows implementation above, too, although we should probably wrap it in a try/finally block to ensure that fill! is called... we certainly need to manually manage any buffers here.

Also, getpass has the nice feature of displaying a snazzy "key" icon in the prompt, at least on MacOS:

😏