Implement GNU jobserver posix client support

[mcprat]

a rework of #2450 supporting all versions of GNU Make, but without Windows support
(I'm not able to test for Windows, and I have doubts with proposed Windows support)

resolves #1139 for posix systems

thanks to @hundeboll for much of the work with this newer implementation

ping @jhasse @digit-google

significant differences:

• no changes to any function parameters
• no new intermediary functions
• instantiate client support in real_main() instead of Plan
• pass references pointers to the jobserver class into other classes
• use a constructor to initialize jobserver client
• release all tokens on any fatal error
• calculate a value for "load capacity" instead of returning SIZE_MAX
• supports both fifo and simple pipe file descriptors from Make
• detect invalid or closed pipe and inform user about the most likely reason

[mcprat]

I forgot I have to adapt to a windows build even if I'm not going to support it on windows...

[mcprat]

The CI for Windows is happy now, but it would be nice to have a tester for Windows...

[mcprat]

I want to point out this concept and ask whether or not this usage of an int is too unusual or non-standard.
This coincides with the last line of the constructor Jobserver::Jobserver() and the value of capacity in CanRunMore() using the absolute value function.

It's pretty easy to rework this. I just happened to have this idea first, to use the token number in place of Enabled() when there actually are no tokens yet.

[mcprat]

found and fixed some minor mistakes, added some commit tags

[mcprat]

sorry I didn't realize that there were "builder" constructors for the test suite, I built and ran the test suite this time.

[mcprat]

oops... I went too fast and everything is still "green" on segfault...

[jhasse]

We're very wary of changes that increase the complexity of Ninja, so a PR that implements both methods while one of them is technical superior and results in less code in Ninja (and to my understand that's the case for fifo), is very unlikely to get merged.

[mcprat]

The current previous struggle is regarding the creation of the Jobserver object in real_main(), nothing to do with the new files and new functionality. I was trying to avoid making functions that call Jobserver functions through another class and instead pass references to the object wherever it's needed, but I can always go back to the other way of creating the object within the Plan struct.

[mcprat]

ah, I see what you mean, it errored on readability...

but the other thing I have doubts about

Run ctest -C Release -vv
CMake Error: Unknown argument: -vv
CMake Error: Run 'ctest --help' for all supported options.
Error: Process completed with exit code 1.

[mcprat]

the bool now defaults to false, and set true with a simple if instead of a ternary, like in the rest of the project 👍🏼

[mcprat]

I believe that all the minor issues caught by the CI are handled now...

[mcprat]

some simplification:

I was reading the Google Style Guide and saw this

...we never allow non-­const reference parameters.

so changes in the last push are:

1. Converted all new references to pointers, to comply with the style guide for Jobserver being a non-const object.

Then I realized that I no longer need to create a Jobserver object for build_test.cc (at least in this commit), so

2. Pass NULL for Jobserver* in Builder instantiations in build_test.cc, check for null dereferencing Jobserver in build.cc

then finally, another opportunity to save lines:

3. Create Jobserver object in NinjaMain instead of real_main() since I realized that only 1 NinjaMain is created in a single process run.

[mcprat]

updated commit message

[mcprat]

@jhasse can we run the CI workflow again?

[mcprat]

small organization update:

• comment rewriting
• style formatting
• save some more lines
• line wrapping
• moved new const functions to header

[digit-google]

Please move all parsing logic to a separate static method that can be unit-tested properly. Also be aware that the GNU Make documentation states explicitly: Be aware that the MAKEFLAGS variable may contain multiple instances of the --jobserver-auth= option. Only the last instance is relevant. Hence this should be implemented properly (and tested).

[digit-google]

Btw, see also to implement the following requirement:

Your tool may also examine the first word of the MAKEFLAGS variable and look for the character n. If this character is present then make was invoked with the ‘-n’ option and your tool may want to stop without performing any operations

From https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html

[mcprat]

Btw, see also to implement the following requirement:

Your tool may also examine the first word of the MAKEFLAGS variable and look for the character n. If this character is present then make was invoked with the ‘-n’ option and your tool may want to stop without performing any operations

From https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html

I wish they didn't mention this in that part of the manual. It's completely out of scope for the jobserver. The jobserver client should not be responsible for determining whether ninja does a "dry run" even though it happens to be parsing MAKEFLAGS which can have many different flags for many different reasons...

This should be implemented in a separate commit (and separate PR) and directly with the BuildConfig object instead of involving the Jobserver objects. Traditionally, ninja does not rely on the environment for any of it's configuration flags, so that's another conversation as well.

[digit-google]

nit: isblank() is locale-dependent, which leads to surprises and is generally slow. It is easier to just compare with ' ' and ' t' in this case.

[digit-google]

I would recommend splitting this PR into multiple commits, i.e.:

1. One that adds the Jobserver class, and its Posix implementation + appropriate unit-tests for it. Also try to make the class as independent from the rest of Ninja as possible (e.g. to not call Warning() or Info() in the parser function, leave that to clients).

2. One that adds usage of the class to build.cc / ninja.cc.

[mcprat]

I would recommend splitting this PR into multiple commits, i.e.:

In my opinion, the first commit in should be functional so that other projects can easily pull the patch while waiting for a release, and also for the "bisect rule", so each individual checkout is functional on it's own. I'm planning on Windows implementation and tests to be separate commits while trying to keep this one small (except for the new files)...

[digit-google]

Is there a way to fallback gracefully to the usual mode when this happens instead?

In general, it is better to avoid calling Fatal() in methods like these, because these conditions cannot be properly unit-tested.

[mcprat]

the remaining calls to Fatal() are for the following cases:

1. MAKEFLAGS from environment told ninja that a fifo object needs to have file descriptors open for it. The open() syscall is ran and errored.
2. File descriptors have been opened to the fifo object, but reading from it has failed not due to blocking. This likely means that the fifo object has been deleted while we have file descriptors that point to nothing, or maybe a problem with the filesystem itself.
3. File descriptors have been opened to the fifo object, but writing to it has failed, again suggesting that the fifo object has been deleted while we have file descriptors that point to nothing or a problem with the filesystem.

They can become warnings, but I think they protect execution from continuing when something is extremely wrong. It should be exceedingly rare for the program to step into these Fatal() calls and they are probably out of scope for unit tests anyway, but I'll let you decide.

[mcprat]

big update...

summary:

• the destructor has been removed. I realized that it never gets run at all. Also, it is not the responsibility of the client to close the pipe. Emergency return of tokens before exiting still happens during call to Clear().
• the constructor has been split into a Parse() function for the majority of the parsing again.
• the Jobserver class is split into base and derived classes, reducing preprocessor #if's to 1.
• Enabled() uses jobserver_closed_ instead of using jobserver_closed_ to put fake FDs.
• macros for constant strings have been converted to static constexpr's as suggested.
• removed underscores from local variables that were previously member variables.
• moved <vector> include from the header to the source.
• small rewrites to comments and warnings.

[mcprat]

smaller update, this might be considered mergeable now.
hopefully all major issues are handled so we can focus on style and nitpicking...

summary:

• now handling and storing the actual token character instead of just a count. this allowed for some simplification so there is a line decrease. this is written assuming a char is 1 byte.
• added a case to fallback to non-parallel build, when the token server provides an FD to read tokens from that is blocking (Make versions 4.2.1 and earlier).
• some light cleanup and rewording
• tested all the way back to Make 4.0, which is when Windows support for jobserver was added. anything earlier than that can be considered "ancient" enough to ignore...

[digit-google]

Please explain what the pointer being passed as argument here should point to? E.g. is it ok to call this with a pointer that points to a value that was never the result of a previous Acquire() code (as suggested by your Clear() function implementation). In which case, what is the default value to be used (also apparently \0 from the code, but you should make that clear in the documentation). Also will the function modify the pointed value or not?

Ideally, users of the API should not have to guess these details by looking at the source code.

I suggest writing a dedicated move-only Token class, even if trivial, to better encapsulate these semantics.

/// A wrapper for token values acquired or released to the pool.
/// A default instance has no value, and is used to indicate that
/// no token is available.
struct Token {
  // Default constructor builds a value-less token.
  Token() = default;
  
  // Explicit constructor for a Token with a value from the pipe.
  explicit Token(uint8_t value) : value_(static_cast<int>(value)) {}
  
  // Move operations are allowed.
  Token(Token&& other) noexcept : value_(other.value_) { other.value_ = -1; }
  Token& operator=(Token&&) noexecpt = default;
  
  // Copy operations are forbidden.
  Token(const Token&) = delete;
  Token& operator=(const Token&) noexcept = default;
  
  /// Returns true if this instance contains a value received from the pipe.
  bool HasValue() const { return value_ != -1; }
  
  /// Return underlying value. It is a runtime error to call this method
  /// if HasValue() returns false.
  uint8_t  GetValue() const {
    assert(HasValue());
    return static_cast<uint8_t>(value_); 
  }
  
  int value_ = -1;
};

Then you can have Acquire() return a Token by value, and Release() take a token by value, and forget about pointers entirely, e.g.:

/// Try to acquire a token from the pool. A value-less token instance is returned
/// if no token is available.
virtual Token Acquire() { return Token(); }

/// Release a previously acquire token to the pool. Does nothing if the
/// token argument has no value.
virtual void Release(Token token) {}

[mcprat]

I rewrote comments in the header to reflect the change in functionality and new returns and arguments.

Making a whole class just to manage the value of the tokens seems complicated and overkill to me. It makes the function declarations read better, but the Token class itself is not very readable and adds a lot of object constructs for not much benefit...

Is there any benefit to avoiding pointers here? or avoiding char?

What if we just made a simple typedef for the tokens?

I'm very confident that a jobserver will never return a NUL char unless it's to indicate "no tokens available" just as I am using it to mean. I'm also pretty confident that Ninja will never receive anything other than a '+' when being used with Make, probably even with forks of Make...

[mcprat]

applied some review comments

summary:

• moved new Clear() function to the RealCommandRunner class as ClearJobTokens()
• ClearJobTokens() function uses range-based for loop statement
• ClearJobTokens() function takes a const reference for the vector
• removed headers for vectors and Edge class from jobserver.h
• reworded comments in jobserver.h

[mcprat]

• Add an std::unique_ptr<Jobserver> jobserver_ member to this class, and default-initialize it with a base/null instance that does nothing. This avoids modifying tests that don't care about the feature at all.
• Add a SetJobserver(std::unique_ptr<Jobserver> jobserver) method to replace the instance's jobserver with a new value. This can be called after constructing a Builder instance where it is needed only. This passes ownership to avoid lifecycle management issues.

nit: isblank() is locale-dependent, which leads to surprises and is generally slow. It is easier to just compare with ' ' and ' t' in this case.

I suggest writing a dedicated move-only Token class, even if trivial, to better encapsulate these semantics.

@digit-google for these comments from you, would you consider any of them critical that I must do? Or perhaps keeping things simple may be better at this point?

Presence of #ifdef _WIN32 is down to 1, and I don't think that last one can be removed, or can it?

[mcprat]

• use calls to Enabled() more instead of rewriting checks

[digit-google]

Oh, I realize I was completely wrong in one of my earlier comments where I stated that Ninja would not pass file descriptor to command sub-processes. It actually totally does, so using --jobserver-auth=R,W works fine on Posix. I tested that actually with this patch. Sorry for the noise.

(Turns out that posix_spawn() doesn't close heritable file descriptors as I mistakenly thought).

[mcprat]

No problem, I had to double-check because I wasn't sure myself, but at least documentation on file descriptors is pretty clear. There is a lot of confusion about how O_CLOEXEC is used by Make, but it has nothing to do with the external pipes.

[mcprat]

• added const to makeflags variable again

[neheb]

these loops look like they're missing break;

[mcprat]

lack of break is intentional, so only the last matching value is the one that takes effect

[neheb]

Sounds like it would be better to reverse the loop then.

[mcprat]

sure it can be reversed, but since this will only be run once per execution of ninja it's not performance critical, so I lean towards the simplest way to write it with less lines and no brackets, etc...

[neheb]

I wonder if emplace_back and no cast is better here.

[mcprat]

I'm not sure exactly what the difference is, I would have to read about it...

the cast is not strictly necessary, it's just protection against invalid characters that I ran into while reading about tokenizing strings.

I saw something about using a std::vector<uint8_t> instead of std::string but that might be confusing...

[mcprat]

apparently after optimization, push_back and emplace_back are exactly the same, unless you are doing fancy constructors for the elements in the vector.

[mcprat]

... no cast is better here.

@neheb I got it from the example here

https://en.cppreference.com/w/cpp/string/byte/isblank

[mcprat]

@digit-google @neheb would this be better? to make the range explicitly equal to isgraph():

  // Tokenize string to characters in flag, then words in flags.
  while (flag_char < strlen(makeflags)) {
    while (flag_char < strlen(makeflags) &&
           ' ' < makeflags[flag_char] && makeflags[flag_char] <= '~') {
      flag.push_back(makeflags[flag_char]);
      flag_char++;
    }

...
...