Introduce a new tool: ninja -t compdb-targets
#2497
Conversation
|
squash to 1 commit to get rid of the merge commits and combine the original with your rename, add yourself to the tags at the end of the original commit it would be nice if we can find a way to get rid of I still think the remaining arguments should not require a long option at all or otherwise |
|
@mcprat Agreed on squashing/rebasing, and i will look into the as for the command-line syntax: i wrote about this in #2319 (though i forgot to tag you there; sorry!), but it would be good to have it inline here too: You wrote:
Prior to the commits in this PR, the output of so
but that means the syntax would become: ...which means that users will need to remember that How would you feel about a short option instead of a long option? (meaning: Also: if we go with an option, are you ok with making the target names comma-delimited? (i think the main reason to use commas is so that we stick to the format expected by Fwiw, in bash 4.2 and later, you can join an array with commas using the printf builtin like so: |
|
@mcprat oh wait, when you wrote:
are you proposing something like: ...or maybe: ...? |
|
@mcprat btw, i assumed that we don't want to break the pre-existing Do we want to make the breaking change of removing the |
|
@mcprat i just thought of another way we could do this: up until now, i've been stuck thinking that ...where the ...which, although it would add an option, it would at least mean that targets are separated by spaces as you requested. (In this case, i would have a weak preference for the long form ( |
|
@JamesWidman I'm aware of your replies in the other PR. I was reiterating, but also clarifying that neither a short or long option would be the ideal (closer to POSIX-like behavior), because it seems like what I said previously was interpreted as only preferring short option or All of this definitely clears things up, it is more complicated than I thought... Here's an idea... what's stopping the parsing stage from checking both the list of targets and list of rules for each argument? Without having looked into it, there's probably different functions used depending on whether the argument is a rule or a target. Maybe each argument can be passed to both, a failure can be caught, and if only one has a problem that's expected, but both failing (or even if neither are failing) would indicate an actual problem. |
|
or rather, since this is a printing-only tool, it can just ignore any errors... |
@mcprat I am trying to understand:
(I do plan on getting to your other points, but first, i really want to understand the answers to these questions.) |
it's not really an option but rather more input, as it doesn't adjust existing outputs but rather just adds to it. consider the way the manual is written right now
instead of now having to describe that there's different ways that one adds a rule to the list vs how one adds a target to the list, it would be nice if the manual can just say:
I consider the design of POSIX has a section called "Utility Syntax Guidelines" that's used to form the standard syntax for |
|
@mcprat First: i think my confusion might be shrinking; thank you! i didn't quite understand what you meant here:
...but i think i understand this part:
so... if i understand you correctly, it sounds like you're saying something like: Is that about right? (If so: i see that the consistency that you're asking for could make it easier for a user to transition from using rules to using targets or vice-versa, and that would end the confusion that prompted my first question above.) Second:
i started reading Issue 8 (IEEE Std 1003.1™-2024 Edition) as well as a draft from 1991, and i just realized that you might be targeting some other version in between (one that is old enough that it is implemented by all the platforms the ninja project wants to run on, but new enough that it specifies all the OS features that the project wants ninja to depend on). i checked so: what version of POSIX should we be reading/targeting? Also: would it make sense to add a mention of that version of POSIX to the wiki and/or |
src/ninja.cc
Outdated
| @@ -945,6 +948,32 @@ void printCompdb(const char* const directory, const Edge* const edge, | |||
| printf("\"\n }"); | |||
| } | |||
|
|
|||
| struct DependentEdgeCollector { | |||
There was a problem hiding this comment.
Please move this class definition to src/graph.h and provide proper documentation + unit-tests for it. See InputsCollector there for inspiration.
I also recommend to make depend_edges a member of the class, and add a method like TakeResult() to move the final result out of the instance one all root targets have been visited.
There was a problem hiding this comment.
Please move this class definition to
src/graph.h
is it ok if i move it to a new header file instead? It's very compdb-specific, and only used by ninja.cc and graph_test.cc, so it would be nice if changing it didn't cause a lot of unrelated source files to be recompiled
src/ninja.cc
Outdated
| std::vector<Edge*> user_interested_edges; | ||
| DependentEdgeCollector deps_edge_collector; | ||
|
|
||
| while (std::getline(iss, target, ',')) { |
There was a problem hiding this comment.
Use SplitStringPiece() from string_piece_util.h instead for simpler and smaller generated code. Use std:: prefixes for all standard types, as we are trying to move away from using namespace std. E.g.:
for (StringPiece target_piece : SplitStringPiece(optarg, ',')) {
std::string err;
std::string target = target_piece.AsString();
Node* node = CollectTarget(target.c_str(), &err);
...
}
src/ninja.cc
Outdated
| return 1; | ||
| } | ||
| if (!deps_edge_collector.CollectFrom(node, &user_interested_edges)) { | ||
| return 1; |
There was a problem hiding this comment.
Silent failure without a message explaning what's going on is a bad user experience :-/
Besides, it looks like your CollectFrom() implementation cannot return false, so get rid of the return value and the condition check here.
src/ninja.cc
Outdated
| @@ -945,6 +948,32 @@ void printCompdb(const char* const directory, const Edge* const edge, | |||
| printf("\"\n }"); | |||
| } | |||
|
|
|||
| struct DependentEdgeCollector { | |||
| bool CollectFrom(Node* node, std::vector<Edge*>* depend_edges) { | |||
There was a problem hiding this comment.
nit: use const Node* for the first argument :-)
|
We do really not care about following POSIX guidelines or imitating the GNU Make design. As such, changing how non-option arguments to It is ok to introduce new options, even if they change the tool's behavior, as long as everything is properly documented, and there is no place for ambiguity. If this is not possible, or makes the UI too cluncky, introducing a new tool might be the better option. |
|
In this specific case, using |
@digit-google Works for me, but is there consensus among maintainers on this point? (I got the sense from @mcprat that some POSIX-ness and Make-ness was desired. I don't feel a strong preference either way, but it would help to have unambiguous direction.)
[thumbs up]
So for example (and please correct me if i'm wrong):
[edit: ...or would item (2) here be viable because it uses a new option that changes the tool's behavior?]
Sounds good to me!
... and by "tool" you mean a new sub-command, right? For example, maybe like: |
|
Yes, except for 2, since Personally, I do not care about sticking to Posix guidelines. Ninja is supposed to be cross-platform anyway, and there are plenty of command-line parsers (like python I also forgot to ask for a regression test for the tool in |
|
@digit-google btw, what's our C++ version for compiling ninja? (i grepped for i see uses of i ask because i wanted to use a structured binding in a place where i thought it would help to add clarity, but that's a C++17 feature, which would require setting Note, according to the clang website:
...and likewise, GCC 11 defaults to C++17. So if we're not going to depend on C++14 or C++17 features, we may want to set |
|
IIRC, the goal is to have the core Ninja sources building with C++11 support only, in order to allow building the tool on older systems (in particular with However, the tests now rely on GTest which mandates C++14 though (and soon C++17), which is why CXX_STANDARD is not applied to them in our I think these requirements were in the top-level I wouldn't mind switching to C++17 personally, but @jhasse is the authority on this specific issue (and this may require updating some of our CI recipes). |
|
@digit-google @jhasse this is a little off-topic for this PR, but: what's our policy on GNU core-language extensions)? some pros/cons (in case this hasn't come up before): pros:
con:
|
|
I believe right now we absolutely want to build with the MSVC toolchain, and support non-GNU and non-Clang toolchains, so these are a deal breakers. |
so it turns out that this is doable, but it requires using at the moment i'm working on regression tests and documentation, but here's a preview of the argument parsing routine (for early feedback on the general shape of this approach): (Before my next push, i will review the coding style guidelines, so hopefully that push will not contain e.g. any naming style violations that might appear below.) Compdb parseCompdbArguments(int argc, char* argv[]) {
Compdb ret;
//
// grammar:
// ninja -t compdb [-hx] [rules] [--targets targets]
//
// in phase 1 of argument parsing, we process [-hx] and "--targets". The
// pseudo-option "--targets" is conceptually more like "--" than an option.
// Traditionally, "--" means "every argument that follows is an operand [i.e.,
// a non-option argument]". Here, "--targets" means "every argument that
// follows is grammatically an operand, and semantically a target. Also, rule
// operands may precede this argument."
//
// If getopt_long returns an instance of "--targets", we store its `optind`
// (see `man 3 getopt`) as the starting index for the list of targets, and we
// store the previous `optind` as the starting index for the list of rules.
//
// in phase 2 of argument parsing, we process the list of rules (if any) and
// the list of targets (if any).
/* [...] */
return ret;
}...where struct Compdb {
enum class Action {
display_help_and_exit,
emit_all_commands,
emit_userSpecified_commands
};
Action action;
EvaluateCommandMode eval_mode = ECM_NORMAL;
typedef std::vector<StringPiece> ArgList;
ArgList targets;
ArgList rules;
};...and the usage looks like this: int NinjaMain::ToolCompilationDatabase(const Options* options, int argc,
char* argv[]) {
Compdb compdb = parseCompdbArguments(argc, argv);
switch (compdb.action) {
/* [...] */
}
return 0;
}
Feedback welcome! (Note: the comment about "storing |
I did originally read the section from Issue 7, but I don't see anything different from Issue 8
I'm not bringing up POSIX-ness or Make functionality to make things harder for us, it should actually make things easier. I understand Ninja does not have POSIX-compliance as a major goal, but I believe it should be done whenever it is easy to do so. POSIX likely recommends against an option having multiple arguments because it's simply not efficient, and doesn't match what most consider to be an "option" compared to what one would define as "input". I personally don't see why we need the secondary stage of imagine instead of
or
we can just have
where the order of operands affects the order of output, but not the actual total content (amount of JSON elements). Then in the manual you don't have to explain a new "option", rather it can simply read: I'm going to try to get that method to work myself, to make sure I'm not preaching for something that isn't possible to begin with... I wanted to have it done already but I've been busy... |
|
Sry misclicked. Regarding C++ standard: I'm okay with everything that works relativly easy on the oldest supported RHEL version (8 currently). I think it would be best to stick with a command line way that would also work easily with other command line parsers. |
|
@mcprat , what you propose is ambiguous, and I assure you that someone, somewhere is going to have a rule and a target with the same name, and won´t understand why they cannot use the tool as expected. There needs to be a explicit separation in the parsing format. Or simply introduce a new tool name for targets, nobody will ever get confused. |
|
Regarding C++, it looks like we should be able to switch to C++17 (to be performed in a separate CL of course). this page says RHEL ships with GCC 8.x or 9.x, and that page says that all C++17 features are supported starting with GCC 8 (and even GCC 7 if you don't care about full support of template deductions). Similarly, this page shows that VS 2019, which we currently use in CI, supports C++17 (unlike VS 2017 which still lacks some features). |
src/ninja.cc
Outdated
| if (!success) { | ||
| Error("cannot determine working directory: %s", strerror(errno)); | ||
| return 1; | ||
| } | ||
|
|
||
| putchar('['); | ||
| for (vector<Edge*>::iterator e = state_.edges_.begin(); |
There was a problem hiding this comment.
Since we're touching this part of the code, can I ask you to change this to use a for-range based loop:
for (const Edge* edge : state_.edges_) {
...
}
src/ninja.cc
Outdated
| case 'h': | ||
| default: | ||
| ret.action = Compdb_Targets::Action::display_help_and_exit; | ||
| goto fin; |
There was a problem hiding this comment.
nit: Just return ret here.
JamesWidman
force-pushed
the
compdb-target
branch
from
1986b5b to
6a7cf8c
Compare
|
(note, |
misc/output_test.py
Outdated
| @@ -12,14 +12,50 @@ | |||
| import tempfile | |||
| import unittest | |||
| from textwrap import dedent | |||
| from typing import Dict | |||
| from typing import Dict, Optional | |||
| from collections.abc import Callable | |||
There was a problem hiding this comment.
oops; i forgot to remove the import of Callable; this will be gone in the next push.
There was a problem hiding this comment.
Superior style for avoiding this:
import os
import sys
import tempfile
import foobar
import typing as T
def func(args: T.Optional[T.Dict[str, str]]) -> T.Callable[........]:
passA commonly used module where the symbols in use are regularly in flux is a strange time to spend most of your time micromanaging which symbols are currently in scope. But everyone does it. I'm always baffled.
The semantic difference between the two is trivial:
from x import yfills up the local namespace and in return accessing the "y" symbol avoids having to do a name lookup inside "x"
The effects on code readability are considerable:
- with
from x import yyou lose the ability to clearly reason at a glance, that many symbols are part of the same logical context
There was a problem hiding this comment.
done and pushed; thank you!
(as i mentioned upthread, i have not touched python in years, so i'm grateful for any python help!)
There was a problem hiding this comment.
Well, like I said an awful lot of people seem to do it so you're not exactly in bad company. :P
Probably the rationale is that from imports contribute less to line max width. That is the usual reason to use from imports for meaningful runtime code.
And if the alternative was truly
import typing
def func(args: typing.Optional[typing.Dict[str, str]]) -> typing.Callable[........]:
passI'd perhaps concede. That's a 7 char cost per symbol and typing symbols always appear on a line where something else is already present. But T is quite distinctive and makes sense to reserve for a language level feature in a way that frobnicator.core.utils as U doesn't really make sense. 🤷 I wish they'd named it "T" all along.
|
is there a recommended way to use |
JamesWidman
force-pushed
the
compdb-target
branch
from
6a7cf8c to
f1f30e3
Compare
misc/output_test.py
Outdated
|
|
||
| default_env = dict(os.environ) | ||
| default_env.pop('NINJA_STATUS', None) | ||
| default_env.pop('CLICOLOR_FORCE', None) | ||
| default_env['TERM'] = '' | ||
| NINJA_PATH = os.path.abspath('./ninja') | ||
|
|
||
| def cook(raw_output: bytes) -> str: |
There was a problem hiding this comment.
may I suggest a more meaningful name, e.g. remove_non_visible_lines here?
src/ninja.cc
Outdated
| // Phase 1: parse options: | ||
| optind = 1; // see `man 3 getopt` for documentation on optind | ||
| int opt; | ||
| while ((opt = getopt(argc, argv, "hx")) != -1) { |
There was a problem hiding this comment.
nit: I get a warning when cross-compiling locally for Windows:
/work2/github-ninja/src/ninja.cc: In static member function ‘static {anonymous}::CompdbTargets {anonymous}::CompdbTargets::CreateFromArgs(int, char**)’:
/work2/github-ninja/src/ninja.cc:1106:38: warning: ISO C++ forbids converting a string constant to ‘char*’ [-Wwrite-strings]
1106 | while ((opt = getopt(argc, argv, "hx")) != -1) {
| ^~~~
You may want to use a const_cast<char*>("hx") here like the rest of the code does for other getopt() calls. :-/
|
Thank you, the last PR looks good apart from minor nits. @jhasse, could you mark this PR as ready for review so that CI builders start building them. I compiled it locally for Linux and Windows only (cross-compiled with mingw64 only). |
JamesWidman
force-pushed
the
compdb-target
branch
from
f1f30e3 to
be4f1a1
Compare
JamesWidman
force-pushed
the
compdb-target
branch
from
be4f1a1 to
69cac1d
Compare
|
i finally looked at the job logs (: |
@digit-google @jhasse yeah; in the workflow for windows, i see that it successfully builds with MSVC, and that the unit tests completed successfully, but it looks like we don't run |
JamesWidman
force-pushed
the
compdb-target
branch
from
69cac1d to
b872496
Compare
on the other hand, |
|
side note: when i try to build the documentation on macOS, using the version of % which xsltproc
/usr/bin/xsltproc
% ./configure.py
wrote build.ninja.
% ninja manual
[1/1] XSLTPROC doc/manual.html
FAILED: doc/manual.html
xsltproc --nonet doc/docbook.xsl build/manual.xml > doc/manual.html
I/O error : Attempt to load network entity http://docbook.sourceforge.net/release/xsl/current/html/docbook.xsl
warning: failed to load external entity "http://docbook.sourceforge.net/release/xsl/current/html/docbook.xsl"
compilation error: file doc/docbook.xsl line 8 element import
xsl:import : unable to load http://docbook.sourceforge.net/release/xsl/current/html/docbook.xsl
ninja: build stopped: subcommand failed.
%...but if i use the version of libxslt provided by Homebrew (after installing XSLT 1.0 Stylesheets for DocBook with % export PATH="$(brew --prefix libxslt)/bin:$PATH"
% ninja manual
[1/1] XSLTPROC doc/manual.html
%i'm guessing that the homebrew version is checking homebrew-standard paths to find the docbook stuff, whereas the macOS version doesn't... should we mention something about this in
brew install libxslt docbook-xsl
export PATH="$(brew --prefix libxslt)/bin:$PATH"(this would be in a separate PR, of course) |
|
Mentioning it in the README.md would be nice. Detecting the version of xslt at configure time and warning about it would be nicer, imho, but I don't know enough about this library and its dependencies to know if this is simple enough to implement. |
Fixes ninja-build#1544 Co-authored-by: Linkun Chen <lkchen@google.com> Co-authored-by: csmoe <csmoe@msn.com> Co-authored-by: James Widman <james.widman@gmail.com>
The introduction of the entry for `compdb-targets` in the `[horizontal]`
labeled list in doc/manual.asciidoc revealed some display issues in the
left column:
First, the web browser would insert a line break in the middle of the
label `compdb-targets`, so that it looked like this:
compdb-
targets
We fix this by applying the `white-space: nowrap` attribute to the left
column.
After this is fixed, we see practically no space between the end of the
longest label and the beginning of the text in the second column; we fix
this with the `padding-right` attribute.
Finally, we align all labels to the right side of the column so that
there is a consistent amount of horizontal space between the end of each
label and the beginning of the text in the second column.
JamesWidman
force-pushed
the
compdb-target
branch
from
b872496 to
b785947
Compare
|
is it normal for workflows to be re-executed when the content of the most recent push is identical to the previous push? |
unfortunately yes, the CI is not that smart, it operates on whether the commit hash has changed not the content. however, GitHub remembers which hashes passed or not so it can display that when a bad commit is placed on top |
https://github.com/orgs/community/discussions/64405 I fear you will be waiting quite long for a resolution but fortunately the CI here doesn't take 2 hours on each run. |
|
@jhasse @digit-google is there anything else i need to do for this PR? (I thought I fixed or addressed everything, but please let me know if I missed anything!) |
|
This looks good to me, it's up to @jhasse now to accept it. |
|
thank you both! |
This fixes #1544.
Here we use the same commit as in #2319, plus one more commit so that the option is spelled
--targets.Many thanks to @lk-chen for the original work in #1546, and to @csmoe for carrying it forward in #2319!
update 2024-10-04:
The most recent push implements the new tool,
ninja -t compdb-targets.todo:
compdb)