Log output of parallel runs separately and output sequentially

[lu-zero]

Since rustc is run in parallel sometimes two outputs might write over each other with quite annoying results.

Would be great if the output of each command invocation is logged separately and reported at execution end in some fixed order (e.g. alphabetical).

[alexcrichton]

This is unfortunately pretty difficult to do, for a number of reasons:

• Capturing output currently means losing colors
• Losing a "streaming output" can be a deal breaker for long compilations

The good news is that almost all compiler invocations are output-free unless you're likely working on the top-level crate (where errors will show up). To me though it seems that the current situation is worth it due to the colorization + streaming. If we could preserve both of those but synchronize the output I'd love to do so!

[lu-zero]

Passing --color always to rustc would work just fine for that purpose of coloring, while for gcc and clang
-fcolor-diagnostics should do.

I can reliably reproduce the problem by having any kind of error in the code and the issue cargo test btw.

[alexcrichton]

Unfortunately that wouldn't work for Windows because colors work differently there (they're a property of the console, not of the output stream).

[lu-zero]

"And that's why we can't have good things"

I have not enough experience about windows to tell alternatives =/

[ptrcmd]

I am also experiencing this issue
Maybe cargo can use a custom "output stream"(say, a byte stream with custom "bytes" that records color), then redirect the colored message to the original output stream (stdout). I think this method should work both on windows and linux?

[U007D]

Yesterday on IRC, I posted

when I compile, I often get errors in duplicate (https://play.rust-lang.org/?gist=3011713c627c43bb382ae76923a94d9a&version=stable). This is annoying when my error lists get long. How can I get rid of the dupes (short of doing a single-threaded compile, of course)?

A discussion followed, focused on buffering the output from multiple processes as the compilation ran, and this issue was referenced as the open issue relevant to my question. Apologies if this is not the correct issue--please let me know and I will move this comment. (This issue's title seems to prescribe an implementation to an implied problem, rather than being a problem/mission statement like "Stop outputting duplicate errors from parallel builds", so perhaps that is where my confusion comes from.)

I and another participant in the discussion wondered why each compilation thread wouldn't consult a centralized dictionary during compilation, something like so:

Process x:

1. has an error it wants to output to the console, say E0583 for l.23
2. acquires lock for the central dictionary and checks l.23's status: no errors have been reported
3. adds E0583 to l.23's status in the dictionary, and releases the lock
4. outputs E0583 to the console.

Process y:

1. parses l.23 and wants to output E0583
2. acquires lock for the central dictionary and checks l.23's status: E0583 has already been reported
3. releases lock to the central dictionary
4. moves on, without issuing duplicate report of E0583

The above synchronization strategy could be upgraded to a lock-free algorithm, but the actual synchronization mechanism isn't really the key point here--it's that each process can still live-stream its output without any chance of duplicate error reporting; the delay from buffering, and the color output issues are both side-stepped by retaining the live streaming behavior.

I assume an approach like this has already been discussed and discarded, but on the off chance it hasn't, I wanted to suggest it in hopes it might help?

[alexcrichton]

@U007D seems plausible to me! That'd be an issue though for rust-lang/rust rather than Cargo as it's the one actually emitting errors.

[U007D]

Oh, so this is the wrong issue, then! :) I'll open the issue/add the comment there. Great, thanks, @alexcrichton.

[ishitatsuyuki]

I think the color part may be actually easier than you think. ANSI is considered universal by most tools: they use ANSI in Windows pipes where the special terminal APIs are not available, and on POSIX ANSI is the standard.

So what we should do is: pass --color=always unconditionally, buffer them, and parse the output (this is the part that needs time to implement). Then, we feed the parsed output into termcolor again, so it's now translated to the native colors, or discarded if colors are not available.