Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate compile --build-properties flag in favor of new featureful --build-property flag #1044

Merged
merged 5 commits into from
Nov 5, 2020

Conversation

silvanocerza
Copy link
Contributor

@silvanocerza silvanocerza commented Oct 22, 2020

Please check if the PR fulfills these requirements

  • The PR has no duplicates (please search among the Pull Requests
    before creating one)
  • The PR follows
    our contributing guidelines
  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • What kind of change does this PR introduce?

Deprecate --build-properties flag in compile command and add a new --build-property flag

  • What is the current behavior?

The following commands would all fail because of quotes (") in them.

export COMPILE_OPTIONS='-DMY_DEFINE="hello world"'
arduino-cli compile -b arduino:avr:uno --build-properties "build.extra_flags='$COMPILE_OPTIONS'" ./my-sketch

arduino-cli compile -b arduino:avr:uno --build-properties="build.extra_flags=\"-DMY_DEFINE=\"hello world\"\"" ./my-sketch

arduino-cli compile -b arduino:avr:uno --build-properties="build.extra_flags=\"-DMY_DEFINE=\"hello world\"\"" ./my-sketch

arduino-cli compile -b arduino:avr:uno --build-properties='compiler.cpp.extra_flags="-DPIN=2" "-DSSID=\"This is a String\"' ./my-sketch
  • What is the new behavior?

The --build-property is introduced to substitute in the future the --build-properties flag, it supports the use of quotes but has a different syntax.
Different build flags can't be specified using the same --build-property flag, nor you can specify the same build flag on separate --build-property otherwise they'll get overridden.

These are some examples.

  ./arduino-cli compile -b arduino:avr:uno /home/user/Arduino/MySketch
  ./arduino-cli compile -b arduino:avr:uno --build-property="build.extra_flags=\"-DMY_DEFINE=\"hello world\"\"" /home/user/Arduino/MySketch
  ./arduino-cli compile -b arduino:avr:uno --build-property="build.extra_flags=-DPIN=2 \"-DMY_DEFINE=\"hello world\"\"" /home/user/Arduino/MySketch
  ./arduino-cli compile -b arduino:avr:uno --build-property=build.extra_flags=-DPIN=2 --build-property="compiler.cpp.extra_flags=\"-DMY_DEFINE=\"hello world\"\"" /home/user/Arduino/MySketch
  • Does this PR introduce a breaking change?

No.


See how to contribute

ctx.CustomBuildProperties = strings.Split(opts.Get("customBuildProperties"), ",")
ctx.OptimizationFlags = opts.Get("compiler.optimization_flags")
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you remove this one? Is this function unused?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, completely unused. I was digging around trying to understand how to fix the issue and stumbled upon it, noticed it wasn't used and decided to delete since I was there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

command.Flags().StringSliceVar(&buildProperties, "build-properties", []string{},
"List of custom build properties separated by commas. Or can be used multiple times for multiple properties.")
command.Flags().StringArrayVar(&buildProperties, "build-properties", []string{},
"List of custom build properties separated by spaces. Or can be used multiple times for multiple properties.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since if you want to change multiple properties you now need to repeat the flag multiple times I think that the correct description is something along the line of:

Suggested change
"List of custom build properties separated by spaces. Or can be used multiple times for multiple properties.")
"Override a build property with a custom value. Can be used multiple times for multiple properties.")

@silvanocerza silvanocerza changed the title Add support for definitions containing quotes in compile --build-properties flag Deprecate compile --build-properties flag in favor of new featureful --build-property flag Oct 27, 2020
@silvanocerza
Copy link
Contributor Author

I've updated the PR with some changes, I marked the --build-properties flag as deprecated in favor of a new --build-property flag that works a bit differently but supports quotes in defines and similar things.

@ubidefeo
Copy link

This took a while to test, had to look hard for a way to break it but haven't found it yet.
it works as far as I'm concerned

@silvanocerza
Copy link
Contributor Author

This took a while to test, had to look hard for a way to break it but haven't found it yet.
it works as far as I'm concerned

This is a first people! Marking it on the calendar! 📆

I'll be waiting for a review by @cmaglie anyway before merging. 😄

@silvanocerza
Copy link
Contributor Author

Also would love some feedback from @obra, @bxparks and @artynet.
You can find a testable build on here: https://github.com/arduino/arduino-cli/pull/1044/checks?check_run_id=1310376266
There's an Artifacts dropdown menu on the top right with download links for different platforms.

@obra
Copy link
Contributor

obra commented Oct 27, 2020

I’ll do my best to give it a workout today. Does arduino-cli have a written policy about what “deprecation” means?

Some projects, for example, have a policy of deprecations warning for two releases before failing with a message that the facility was removed for another n releases. Some projects explicitly state that every .0 is fair game for breakage.

@silvanocerza
Copy link
Contributor Author

I’ll do my best to give it a workout today. Does arduino-cli have a written policy about what “deprecation” means?

Thanks, appreciate it. :)

Some projects, for example, have a policy of deprecations warning for two releases before failing with a message that the facility was removed for another n releases. Some projects explicitly state that every .0 is fair game for breakage.

We're working on it, we're exploring different alternatives.

@obra
Copy link
Contributor

obra commented Oct 27, 2020

Some projects, for example, have a policy of deprecations warning for two releases before failing with a message that the facility was removed for another n releases. Some projects explicitly state that every .0 is fair game for breakage.

We're working on it, we're exploring different alternatives.

I can respect that answer :) My background comes from managing Perl's backwards-compatibility policy, which is, roughly "do not break on a user unless it is unavoidable." Being that focused on not hurting users comes with a huge maintenance burden that can calcify a project if such a policy is applied to early.

Is there a place that it would be useful for me to comment about what I as a consumer of the tool (and as someone who's been responsible for writing such policies) might want? It's 100% ok if the answer is "feedback is not useful right now"

@rsora
Copy link
Contributor

rsora commented Oct 27, 2020

@obra

I can respect that answer :) My background comes from managing Perl's backwards-compatibility policy, which is, roughly "do not break on a user unless it is unavoidable." Being that focused on not hurting users comes with a huge maintenance burden that can calcify a project if such a policy is applied to early.

Exactly this :) Being the CLI still in alpha and in very heavy development, we are running fast to stabilize the APIs and implement all the features we think are needed. We are trying to avoid to enforce a structured Deprecation policy ATM to not slow down the development.

However we recognize that the project gained traction, and we have users that rely on the CLI in their projects. We don't want to let our users down :) so I started to modify slightly our contribution guidelines here #1039 and our internal release documents in order to start managing the project breaking changes in the easiest way possible.

It would be great to get your feedback on the PR 👍

@rsora
Copy link
Contributor

rsora commented Oct 27, 2020

Let me clarify that having a structured deprecation policy is a milestone in our roadmap for the 1.0.0 stable release, so it is something we are working on.

Maybe my previous message was a bit misleading on this specific point. 😄

@alranel
Copy link
Contributor

alranel commented Oct 27, 2020

@obra, I'm so happy to see you again in this repository :)
Your feedback is very welcome!

@obra
Copy link
Contributor

obra commented Oct 28, 2020

@rsora Is the internal release document something that can't be made public?

Poking at 0.13.. there's nowhere I see in the help or the online docs that arduino-cli is an alpha release. Ditto https://arduino.github.io/arduino-cli/latest

Now, most advanced users can probably figure out that 0.13 isn't a "release product" version number. But it looks like older alphas had 'alpha' in the filename or version.

In terms of something "lightweight" for deprecation policy before 1.0, maybe something like this would be enough?

"arduino-cli is a powerful tool that many people depend on, but it is still pre-release software. We are still hard at work making sure that arduino-cli 1.0 will be a tool that you can trust long into the future. Because of this, we may make backwards-incompatible changes at any time, but promise to do so only after careful deliberation."

Copy link
Member

@cmaglie cmaglie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a note to check the examples on the help

For the rest LGTM

" " + os.Args[0] + " compile -b arduino:avr:uno /home/user/Arduino/MySketch\n" +
" " + os.Args[0] + " compile -b arduino:avr:uno --build-property=\"build.extra_flags=\\\"-DMY_DEFINE=\\\"hello world\\\"\\\"\" /home/user/Arduino/MySketch\n" +
" " + os.Args[0] + " compile -b arduino:avr:uno --build-property=\"build.extra_flags=-DPIN=2 \\\"-DMY_DEFINE=\\\"hello world\\\"\\\"\" /home/user/Arduino/MySketch\n" +
" " + os.Args[0] + " compile -b arduino:avr:uno --build-property=build.extra_flags=-DPIN=2 --build-property=\"compiler.cpp.extra_flags=\\\"-DSSID=\\\"hello world\\\"\\\"\"-DMY_DEFINE=\"hello world\"' /home/user/Arduino/MySketch\n",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you sure that this last example is correct? on the help is displayed as:

  ./arduino-cli compile -b arduino:avr:uno --build-property=build.extra_flags=-DPIN=2 --build-property="compiler.cpp.extra_flags=\"-DSSID=\"hello world\"\""-DMY_DEFINE="hello world"' /home/user/Arduino/MySketch

and it doesn't look right to me...
To simplify your life you may use the golang verbatim string operator (backtick) so the help strings will not require triple quoting:

  `--build-property="build.extra_flags=\"-DMY_DEFINE=\"hello world\"\"" /home/user/Arduino/MySketch`

another small improvement may be to drop the (optional) = sign after --build-property so instead of:

--build-property="build.extra_flags=\"-DMY_DEFINE=\"hello world\"\""

we may write

--build-property "build.extra_flags=\"-DMY_DEFINE=\"hello world\"\""

that is slightly clearer IMHO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right! The last example is plainly wrong, I'll fix it.

Somehow I always forget about the backtick. 🤦

@bxparks
Copy link

bxparks commented Oct 28, 2020

Hello, thank you for working on this issue. This is the main blocker that prevents me from using the ArduinoCLI.

Is there a reference or documentation that explains the tokenization and parsing rules for the expression on the right side of the compiler.cpp.extra_flags=? I would have assumed that the entire string to the right of the = would be passed without any processing to the compiler binary. But after some trial and error, there seems something is doing some processing of double-quotes and white spaces, but I cannot deduce the rules even after many attempts of trial and error.

  1. The original 4th example in the compile --help output contained 2 macros, SSID and MY_DEFINE. There were at least 2 problems with it. One, an unmatched single-quote, which you fixed. Two, a missing space before the second -D. If I attempt to get the original example to work, the following does not work (I'm using the bash(1) shell where single-quoted strings are passed unprocessed to the arduino-cli binary):
$ arduino-cli compile --warnings all --verbose -b arduino:avr:uno \
    --build-property 'compiler.cpp.extra_flags=-D SSID="hello world" -D MY_DEFINE="hello world"'
[...]
.../avr-g++ [...] -D "SSID=\"hello" "world\"" -D "MY_DEFINE=\"hello" "world\"" [...]
avr-g++: error: world": No such file or directory
avr-g++: error: world": No such file or directory
  1. It looks like the the parser is tokenizing the whitespace within the quoted "hello world". Because if we remove the white space within the quoted string, the parsing is correct:
$ arduino-cli compile --warnings all --verbose -b arduino:avr:uno \
    --build-property 'compiler.cpp.extra_flags=-D SSID="helloworld" -D MY_DEFINE="helloworld"'
[...]
.../avr-g++ [...] -D "SSID=\"helloworld\"" -D "MY_DEFINE=\"helloworld\"" [...]
  1. The only way I have found to get around this tokenization problem is the use of unescaped, nested, double-quote (which should not work, but it does):
$ arduino-cli compile --warnings all --verbose -b arduino:avr:uno \
    --build-property 'compiler.cpp.extra_flags="-D SSID="hello world"" "-D MY_DEFINE="hello world""'
[...]
.../avr-g++ [...] "-D SSID=\"hello world\"" "-D MY_DEFINE=\"hello world\"" [...]
  1. If I attempt to double-quote the entire string on the right side of the compiler.cpp.extra_flags=, the parsing fails:
$ arduino-cli compile --warnings all --verbose -b arduino:avr:uno \
    --build-property 'compiler.cpp.extra_flags="-D SSID="hello world" -D MY_DEFINE="hello world""'
[...]
.../avr-g++ [...] "-D SSID=\"hello world" -D "MY_DEFINE=\"hello" "world\"\"" [...]
avr-g++: error: world"": No such file or directory

It seems like nested-double-quoting only works around a single -D flag.

  1. However, the nested double-quote escape breaks down when I try to embed a naked double-quote character inside the SSID, because the parser does not seem to know to skip over the backslashed escaped double-quote. Let's say I want the SSID to be the raw string hello" world. The following does not work:
$ arduino-cli compile --warnings all --verbose -b arduino:avr:uno \
    --build-property 'compiler.cpp.extra_flags="-D SSID="hello\" world"" "-D MY_DEFINE="hello\" world""'
[...]
.../avr-g++ [...] "-D SSID=\"hello\\" "world\"\"" "-D MY_DEFINE=\"hello\\" "world\"\"" [...]
avr-g++: error: world"": No such file or directory
avr-g++: error: world"": No such file or directory

Remarks:

The --build-property flag is definitely better than the previous --build-properties flag. But it seems like the tokenizing and parsing of the -D flags is still not fully accurate. In particular, it does not seem to support string macros with embedded spaces or embedded double-quotes. I think a lot of these problems are caused by multiple levels of tokenizing, and double-quote parsing. If you can clarify and document what those parsing rules are, I think that would help a lot.

I am still of the opinion that exposing the -D flag directly from the arduino-cli go binary is the best way to implement this feature. This would remove a number of confusing layers of parsing, tokenizing and quote and backslash escaping.

Another comment: Some of the compile --help examples show the build.extra_flags= property being overridden. But isn't the build.extra_flags property a dangerous one to override? All of the various 3rd party cores that I've looked at define a non-empty value for the build.extra_flags= property. If the end-user overrode this, it is not clear that things would work properly. The compiler.cpp.extra_flags= seems to be the property that is unused by various 3rd party cores, so can be safely overridden by end-users.

Edit: Fix typos and grammar.

@bxparks
Copy link

bxparks commented Oct 28, 2020

By the way, I used the following Test.ino file to test my arduino-cli commands above:

const char STRING[3] = SSID; // will trigger a compiler warning if SSID is > 2 characters

#ifdef MY_DEFINE
  #warning MY_DEFINE
#endif

void setup() { }

void loop() { }

@silvanocerza
Copy link
Contributor Author

Hello, thank you for working on this issue. This is the main blocker that prevents me from using the ArduinoCLI.

Glad to help. :)

Is there a reference or documentation that explains the tokenization and parsing rules for the expression on the right side of the compiler.cpp.extra_flags=? I would have assumed that the entire string to the right of the = would be passed without any processing to the compiler binary. But after some trial and error, there seems something is doing some processing of double-quotes and white spaces, but I cannot deduce the rules even after many attempts of trial and error.

There's no documentation right now about it, we'll see to add something. The tokenization is actually done in two places, first by the cobra that parses and validates the arguments, then the legacy code of CLI collects all the flags necessary for compilation, linking, etc. and manipulates them again. The second step is kinda obscure to me for now so I'd have to investigate a bit to give you a full answer.

  1. The original 4th example in the compile --help output contained 2 macros, SSID and MY_DEFINE. There were at least 2 problems with it. One, an unmatched single-quote, which you fixed. Two, a missing space before the second -D. If I attempt to get the original example to work, the following does not work (I'm using the bash(1) shell where single-quoted strings are passed unprocessed to the arduino-cli binary):
$ arduino-cli compile --warnings all --verbose -b arduino:avr:uno \
    --build-property 'compiler.cpp.extra_flags=-D SSID="hello world" -D MY_DEFINE="hello world"'
[...]
.../avr-g++ [...] -D "SSID=\"hello" "world\"" -D "MY_DEFINE=\"hello" "world\"" [...]
avr-g++: error: world": No such file or directory
avr-g++: error: world": No such file or directory

That was actually a mistake by my side, there was an extra define that should have not been there, the current 4th example is this one:

compile -b arduino:avr:uno --build-property build.extra_flags=-DPIN=2 --build-property "compiler.cpp.extra_flags=\"-DSSID=\"hello world\"\"" /home/user/Arduino/MySketch
  1. It looks like the the parser is tokenizing the whitespace within the quoted "hello world". Because if we remove the white space within the quoted string, the parsing is correct:
$ arduino-cli compile --warnings all --verbose -b arduino:avr:uno \
    --build-property 'compiler.cpp.extra_flags=-D SSID="helloworld" -D MY_DEFINE="helloworld"'
[...]
.../avr-g++ [...] -D "SSID=\"helloworld\"" -D "MY_DEFINE=\"helloworld\"" [...]

This can be avoided by wrapping each define in quotes like this:

--build-property 'compiler.cpp.extra_flags="-D SSID="helloworld"" "-D MY_DEFINE="helloworld""'

This is necessary because the second step that I was talking about above splits the tokens if they're not wrapped.

  1. The only way I have found to get around this tokenization problem is the use of unescaped, nested, double-quote (which should not work, but it does):
$ arduino-cli compile --warnings all --verbose -b arduino:avr:uno \
    --build-property 'compiler.cpp.extra_flags="-D SSID="hello world"" "-D MY_DEFINE="hello world""'
[...]
.../avr-g++ [...] "-D SSID=\"hello world\"" "-D MY_DEFINE=\"hello world\"" [...]

This is actually the correct way to use this flag.

  1. If I attempt to double-quote the entire string on the right side of the compiler.cpp.extra_flags=, the parsing fails:
$ arduino-cli compile --warnings all --verbose -b arduino:avr:uno \
    --build-property 'compiler.cpp.extra_flags="-D SSID="hello world" -D MY_DEFINE="hello world""'
[...]
.../avr-g++ [...] "-D SSID=\"hello world" -D "MY_DEFINE=\"hello" "world\"\"" [...]
avr-g++: error: world"": No such file or directory

It seems like nested-double-quoting only works around a single -D flag.

Same as above, you must wrap each define.

  1. However, the nested double-quote escape breaks down when I try to embed a naked double-quote character inside the SSID, because the parser does not seem to know to skip over the backslashed escaped double-quote. Let's say I want the SSID to be the raw string hello" world. The following does not work:
$ arduino-cli compile --warnings all --verbose -b arduino:avr:uno \
    --build-property 'compiler.cpp.extra_flags="-D SSID="hello\" world"" "-D MY_DEFINE="hello\" world""'
[...]
.../avr-g++ [...] "-D SSID=\"hello\\" "world\"\"" "-D MY_DEFINE=\"hello\\" "world\"\"" [...]
avr-g++: error: world"": No such file or directory
avr-g++: error: world"": No such file or directory

This is an interesting case that I didn't think about, I'll see if it's possible to implement it.

Remarks:

The --build-property flag is definitely better than the previous --build-properties flag. But it seems like the tokenizing and parsing of the -D flags is still not fully accurate. In particular, it does not seem to support string macros with embedded spaces or embedded double-quotes. I think a lot of these problems are caused by multiple levels of tokenizing, and double-quote parsing. If you can clarify and document what those parsing rules are, I think that would help a lot.

I am still of the opinion that exposing the -D flag directly from the arduino-cli go binary is the best way to implement this feature. This would remove a number of confusing layers of parsing, tokenizing and quote and backslash escaping.

I've not investigated the addition of the -D flag, but probably we'd still have the same issues we have with --build-properties or --build-property depending which type we tell cobra to use for that variable.

Another comment: Some of the compile --help examples show the build.extra_flags= property being overridden. But isn't the build.extra_flags property a dangerous one to override? All of the various 3rd party cores that I've looked at define a non-empty value for the build.extra_flags= property. If the end-user overrode this, it is not clear that things would work properly. The compiler.cpp.extra_flags= seems to be the property that is unused by various 3rd party cores, so can be safely overridden by end-users.

I don't know about this, probably someone with more knowledge about this can give a better answer about it. :)

@silvanocerza silvanocerza force-pushed the scerza/fix-compile-build-properties branch from f68c808 to e3507b4 Compare November 5, 2020 09:41
@silvanocerza
Copy link
Contributor Author

Force pushed to resolve conflicts, merging after checks are sucessful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Can't include quoted strings in --build-properties args
7 participants