Not reading complete output from child process on windows.

[yuyichao]

See some discussion in #10984 and CI result for (32bit and 64bit).

The symptom is that the output captured from a child with redirected STDERR is truncated. Since I don't have a machine to reproduce it locally, I don't know if the problem is in the child process or in the parent.

[tkelman]

Thanks for opening (I should've just done it myself, sorry for the trouble)

Here's some local results from your branch

 _/ |\__'_|_|_|\__'_|  |  (detached from yuyichao/ambiguous-warning)/ac825f4* (fork: 1 commits, 0 days)
|__/                   |  x86_64-w64-mingw32

julia> redir_err = "redirect_stderr(STDOUT)";
julia> exename = joinpath(JULIA_HOME, Base.julia_exename());
julia> script = "$redir_err; f(a::Number, b...) = 1;f(a, b::Number) = 1";
julia> warning_str = readall(`$exename -f -e $script`)
"Warning: New definition \n    f(Any, Number) at none:1\nis ambiguous with: \n    f(Number, Any...) at none:1.\nTo fix, define \n    f(Number, Number)"

julia> script = "$redir_err; module A; f() = 1; end; A.f() = 1";
julia> warning_str = readall(`$exename -f -e $script`)
"Warning: Method definition f"

[yuyichao]

Actually I'm interested to see if the problem is still there if the child process (which generate the warning) is not spawn by julia or if the STDERR is not redirected to STDOUT. Maybe also how it looks for the original warning (the one that prints <func_name>Tuple{...} for function signature).

[tkelman]

Useful to note, when run from cmd.exe instead of cygwin mintty, the first test outputs jl_uv_writecb() ERROR: bad file descriptor EBADF.

On your branch, in cmd, I get this for running at the repl instead of a spawned process:

julia>  module A; f() = 1; end; A.f() = 1
Warning: Method definition f() in module A at none:1 overwritten in module Main
at none:1.
f (generic function with 1 method)

in mintty, it looks a little different

julia> module A; f() = 1; end; A.f() = 1
Warning: Method definition f() in module A at none:1f (generic function with 1 method) overwritten in module Main
 at none:1.

[yuyichao]

Is that the same if not run from REPL? I guess a missing flush can only (or at least more likely to) happen when running non-interactively?

[tkelman]

from cmd, your branch:

D:\cygwin64\home\Tony\julia>usr\bin\julia -f -e "module A; f() = 1; end; A.f() =
 1"
Warning: Method definition f() in module A at none:1 overwritten in module Main
at none:1.

from mintty, your branch:

Tony@TK-samsung ~/julia
$ usr/bin/julia -f -e "module A; f() = 1; end; A.f() = 1"
Warning: Method definition f

[tkelman]

The redirection probably has something to do with this

 _/ |\__'_|_|_|\__'_|  |  (detached from yuyichao/ambiguous-warning)/ac825f4* (f
ork: 1 commits, 0 days)
|__/                   |  x86_64-w64-mingw32

julia> redir_err = "redirect_stderr(STDOUT)";

julia> exename = joinpath(JULIA_HOME, Base.julia_exename());

julia> script = "$redir_err; module A; f() = 1; end; A.f() = 1";

julia> warning_str = readall(`$exename -f -e $script`)
"Warning: Method definition f"

julia> script = "module A; f() = 1; end; A.f() = 1";

julia> warning_str = readall(`$exename -f -e $script`)
Warning: Method definition f() in module A at none:1 overwritten in module Main
at none:1.
""

[yuyichao]

Is this

```
julia> warning_str = readall(`$exename -f -e $script`)
Warning: Method definition f() in module A at none:1 overwritten in module Main
at none:1.
""
```

under mintty or cmd.exe?

Edit: and by "compiled with gcc" I mean linked to glibc under windows (whatever it is called, cygwin? mingw32?....).

[tkelman]

It was under cmd. Under mintty, I get

julia> warning_str = readall(`$exename -f -e $script`)
Warning: Method definition f""

[yuyichao]

Updated the title since IMHO this issue is on the reading side but not the writing side.
Since it is not specific to julia but might be an important enough feature, I'm not sure if it is better to just close this and report upstream (whoever it is) or try to find a workaround....

[tkelman]

I think we might be mixing up streams somewhere, or redirection isn't entirely working. There are well-known redirection quirks with mintty that we've mostly been working around okay, and the necessary changes have largely been upstreamed to libuv by now. This might be something else, but it could just as easily be a bug in Julia's code somewhere.

[tkelman]

glibc under windows (whatever it is called, cygwin? mingw32?....)

No such thing as glibc on Windows, we don't link to any posix layer with Windows Julia - cygwin and msys-1.0.dll and msys-2.0.dll don't support asynchronous IO so libuv doesn't work there. We use a posix build environment for bash and make, and the mintty terminal emulator needs one of those posix layers, but the compiled julia exe is using mingw-w64 which links against the non-posix msvcrt.dll C runtime.

[vtjnash]

it looks like redirect_std••• on windows is having trouble with ordering of the output. are the various parts of that warning's outputs going to the same uv_handle_t?

[yuyichao]

@vtjnash I guess you are talking about my pull request.

They are AFAIK. 1) they works on linux. 2) the JL_STREAM* and the jl_value_t* I'm writing to were both used by the original code for this very message. (see here)

Edit; didn't realize you meant uv_handle_t. Not sure about that but at least they are the ones used before.

[vtjnash]

yes, i thought it had been merged. although it is also worth comparing to the old code, although it seems from this thread that the issue is not new. JL_STREAM is a typealias to uv_handle_t. The jl_value_t should have the same handle internally also.

the linux design is arguably somewhat more reliable, since it is significantly simpler while providing more functionality.

[vtjnash]

@Keno could this be related to #12144? (ref #11017 (comment))

[vtjnash]

i think this may be (partially) fixed by JuliaLang/libuv@0ada24a