Provide dedicated sets of "extra flags" properties for platform developers and users

[per1234]

Describe the request

Establish two sets of standard "extra flags" build properties:

• one set for use by platform developer
• one set for use by the user

These should be clearly documented as being dedicated to the exclusive use by either the platform developer or the user.

The properties proposed here are only arbitrary properties, without any special use by the build system, so no changes would required in the build system. It is only a matter of setting an example for 3rd party platform developers by adding these additional properties to the platform.txt files of the official platforms with comments clearly explaining their intent.

Ideally, the properties would be documented in the Arduino platform specification as well (#985).

Describe the current behavior

Arduino has established a convention of providing "extra flags" properties in platform.txt. These properties are intended to allow the compilation process to be customized by the addition of arbitrary flags in the compilation commands.

For example:

https://github.com/arduino/ArduinoCore-avr/blob/1.8.6/platform.txt#L39-L49

# This can be overridden in boards.txt
build.extra_flags=

# These can be overridden in platform.local.txt
compiler.c.extra_flags=
compiler.c.elf.extra_flags=
compiler.S.extra_flags=
compiler.cpp.extra_flags=
compiler.ar.extra_flags=
compiler.objcopy.eep.extra_flags=
compiler.elf2hex.extra_flags=

These properties are referenced in the appropriate compilation recipes in platform.txt (example).

The empty definitions in platform.txt ensure that if the extra flags are not defined elsewhere, the compilation recipes referencing them will still work. Those default empty definitions can be overrriden from any of the other platform configuration files:

• boards.txt
• boards.local.txt
• platform.local.txt
• global platform.txt

or from the command line:

• arduino-cli compile --build-property
• arduino --pref

---

From the comments in the official platform.txt files, it seems that build.extra_flags is intended to be for the use of the boards platform developer, while the other "extra flags" properties are intended to be for the use of the user (because a platform developer has no need for platform.local.txt).

A common use case of these properties for the user would be to add -D flags (e.g., #210 (comment), arduino/Arduino#6344), but there are other uses.

However, platform developers sometimes use these properties in their boards.txt file (example). This presents the problem that the user risks inadvertently overriding the flags defined by the boards platform if they use the "extra flags" properties to modify the compilation command (example).

Arduino CLI version

N/A

Operating system

All

Operating system version

Any

Additional context

Re: Possible Harmful Effect on APIs

On a certain level, this does provide the user the global macro setting capability that has been often requested, and often rejected:

• Include global UserSettings.h file arduino-builder#15
• Adding support for sketch-specific build properties arduino-builder#29
• Read additional BuildProperties from optional ./platform.sketch.txt file arduino-builder#282
• A further generalized idea of modifying UserSettings.txt sketch-wide arduino-builder#283
• Include sketch directory to include search path  arduino-builder#354
• Please add a way to include project wide #defines within the IDE arduino-ide#1837
• Ability to specify additional compiler flags [imported] Arduino#421
• Library file dependent on sketch file not compiling in Arduino IDE Arduino#2547
• Add sketch path to the compile include path Arduino#3635
• Issue #3635 Added the sketch folder to the path Arduino#3717
• Include sketchbook folder and LibrarySettings.h Arduino#3757
• Feature Request - ability to append to build.extra_flags from within main sketch file. Arduino#4528
• Generate global macros according to contents of specially named header file in sketch Arduino#5084
• Add current sketch dir to include path when compile library Arduino#5390
• Provide advanced compilation features Arduino#7189
• Possibility to specify compilation flags to arduino-builder from the IDE Arduino#11108
• Using -D cppflags possible? #159
• Build flags set in my boards.txt file not used in the compilation command #210
• ARDUIFINE: global compiler options extracted from ctags parsing #1117
• libdefs: a way for user to configure libraries #1517
• Use of defines in reproducible builds tooling-rfcs#9
• Source/Header file edits for build profiles. tooling-rfcs#22

However, I think this proposal strikes the right balance of:

• Allowing advanced users to do advanced things
• Not exposing standard users to additional complexity that makes the Arduino development software less beginner friendly
• Being very easy for Arduino to implement

The concern has been that allowing users to set global macros will result in library developers unnecessarily using these as part of the API, making that API less beginner friendly (arduino/arduino-builder#15 (comment)). However, if the user is required to use a command line interface or create a specially formatted configuration file in a difficult to access location, the average user won't be interested, and so it will only make sense for a library developer to use this feature for advanced, rarely used features of the library (e.g., enable debug output for troubleshooting of the library). We have already had this "extra flags" capability since 2014, yet I haven't seen it result in any significant incorporation of user set global macros into the user-level API of Arduino libraries.

Additional requests

• Provide dedicated sets of "extra flags" properties for platform developers and users #846 (comment)
• https://forum.arduino.cc/t/using-build-property-to-set-optimize-and-define-symbols/1254465
• https://forum.arduino.cc/t/compiler-option-defines-overriding-esp32-and-not-compiling/990862
• https://forum.arduino.cc/t/how-to-add-compile-options-to-arduino-ide-1-8-13/692640/3
• https://forum.arduino.cc/t/ide-to-command-line-build-support-d/682481/3
• https://forum.arduino.cc/t/what-could-prevent-platform-local-txt-from-being-read/674998

Related

• esp32-c3 linker errors, project works fine on windows #2706

Issue checklist

• I searched for previous requests in the issue tracker
• I verified the feature was still missing when using the latest nightly build
• My request contains all necessary details

[alranel]

Some additional thoughts:

• Even if we clarify the semantics in the documentation, we'd still have a legacy. Maybe with a simple crawler we can estimate how many existing cores are setting these flags (and which ones). We might consider opening PRs on all existing cores that are setting a flag intended for users.
• We could also print warnings to the console at compile time whenever something is overridden, to help users troubleshoot.
• Those compiler.*.extra_flags seem a bit unfriendly and fragmented for the average user who may just want to add a -DFOOBAR and may be unsure about where exactly should that be put
• As an alternative to all this, could we consider a syntax that allows to append flags? Such as build.extra_flags+=-DFOOBAR

[bxparks]

I think the number one reason for providing an easy -D flag mechanism: passing in secrets that should not be checked into source control. For example:

• WIFI_SSID
• WIFI_PASSWORD
• HTTP_BASIC_AUTH_USER
• HTTP_BASIC_AUTH_PASSWORD
• FROM_PHONE_NUMBER
• TO_PHONE_NUMBER
• EMAIL_ADDRESS

The Arduino IDE should provide a graphical UI for storing these #define values, outside of the source code. The Arduino-CLI should provide a -D flag. It should have taken me 5 minutes. Instead I spend 2 hours searching the internet to learn about the --build-properties flag, and to figure out how to hack it because there is basically no documentation for it.

Not having an obvious mechanism for injecting secrets continues to validate the old joke: "The S in IoT stands for Security".

The number two reason for the -D flag: to pass in different configuration values for different boards. I often compile the same program targeted to different hardware using different microcontrollers (e.g. AVR, ESP8266, SAMD), having different pin numbers, LEDs, sensors, etc.

[per1234]

We might consider opening PRs on all existing cores that are setting a flag intended for users.

My suggestion would be to just submit PRs adding support for the new properties to all boards platforms. It's only a matter of inserting the new properties into the compilation recipes and adding the default property definitions to platform.txt. Using a list of boards platforms I curate for this very type of project, it should be easy to write a script to set everything up for the PR automatically. Of course, a human would still need to do a check of the commit diff before submitting each PR to make sure nothing went awry.

We could also print warnings to the console at compile time whenever something is overridden, to help users troubleshoot.

If it's easy to implement, this information would certainly be useful. At the very least, we should document the override priorities. There are several places properties can be defined and currently no documentation of what the override priorities are (arduino/Arduino#4431).

However, I would definitely not want to see this output exposed to the normal user. Property overrides are a fundamental part of how the platform system works and is used, not something that should be considered cause for a "warning". Perhaps the information could be printed in the Arduino CLI log output at either the debug or trace level. Related: https://github.com/arduino/Arduino/issues/1415

Those compiler.*.extra_flags seem a bit unfriendly and fragmented for the average user who may just want to add a -DFOOBAR and may be unsure about where exactly should that be put

That would be solved by adding a user dedicated equivalent to build.extra_flags (something like build.user_extra_flags). The recipe-specific properties (e.g., compiler.cpp.extra_flags) are for when they need to configure a specific part of the compilation process.

As an alternative to all this, could we consider a syntax that allows to append flags? Such as build.extra_flags+=-DFOOBAR

It's a very interesting idea!

Currently, + is an allowed character for property names, so that line is defining a property named {build.extra_flags+}. I'm not aware of any properties using a trailing +. If it is in use, I'm sure it's not common.

I don't have a good understanding of these configuration files. Are they using an existing format, or is this something created by Arduino? (EDIT: see https://godoc.org/github.com/arduino/go-properties-orderedmap) The .properties extension of library.properties makes me think that file at least is using the Java properties file format (EDIT: it's not Java properties). I think this would be a factor in deciding about adding an append operator.

[ubidefeo]

@per1234
the platform flags implementation is certainly useful and @alranel suggestion of "adding" is an interesting one, we need to look into it.
In lieu of -D could we then pass a set of these to the --fqbn like we do for cpu etc?

e.g.:
arduino-cli compile --fqbn arduino:avr:nano:cpu=atmega328old,build.extra.flags+=
I'm trying to figure out how we'd assign a value to a #define passed in via -D or other mechanism

[per1234]

I think the number one reason for providing an easy -D flag mechanism: passing in secrets that should not be checked into source control.
...

A system for doing this has been established in the Arduino Web Editor:
https://arduino.github.io/arduino-cli/dev/sketch-specification/#secrets

You could simply add the header file containing the secrets to .gitignore to preventing it from being checked in

The Arduino IDE should provide a graphical UI for storing these #define values, outside of the source code.

Although I am aware of some other applications where I do agree that this capability would be useful, I'm still not convinced that the benefits outweigh the harm. I remain open minded on the subject though.

The Arduino-CLI should provide a -D flag. It should have taken me 5 minutes.

Do you think my proposal would be sufficient to make you feel that Arduino CLI is providing the "easy -D flag mechanism" you are requesting?

Instead I spend 2 hours searching the internet to learn about the --build-properties flag, and to figure out how to hack it because there is basically no documentation for it.

I think that's a fair criticism. The command line help format is a bit limiting when it comes to providing detailed documentation on complex subjects like this and the online commands documentation is generated from the command line help.

In this case, I think adding the URL of the platform specification to the help text would be sufficient:

--build-properties strings List of custom build properties separated by commas. Or can be used multiple times for multiple properties. See https://arduino.github.io/arduino-cli/latest/platform-specification/#configuration-files-format for details

What do you think?

The number two reason for the -D flag: to pass in different configuration values for different boards. I often compile the same program targeted to different hardware using different microcontrollers (e.g. AVR, ESP8266, SAMD), having different pin numbers, LEDs, sensors, etc.

We already provide a mechanism for this. A macro is defined for each board according to ARDUINO_{build.board}:
https://arduino.github.io/arduino-cli/dev/platform-specification/#boardstxt

[per1234]

In lieu of -D could we then pass a set of these to the --fqbn like we do for cpu etc?

@ubidefeo it's certainly possible for the platform author to add -D flags according to custom board options. Those custom board options can be set via the FQBN.

For example, a -D flag is set according to the Uno WiFi Rev2's "Registers emulation" option:
https://github.com/arduino/ArduinoCore-megaavr/blob/1.8.3/boards.txt#L36

This is the ideal approach for a platform developer to offer the user the ability to define specific global macros (though typically the user would not need to have any understanding of exactly what that custom board option is doing under the hood).

However, the custom board options system doesn't allow users to pass arbitrary -D flags. As for the idea of using the FQBN to add that capability, my preference is that the existing capabilities of the platform system be used whenever possible, adding new capabilities only when a significant deficiency is discovered. The reason is because adding new capabilities makes the system harder to maintain and harder to learn.

The existing system is pretty flexible, so I think it's possible to do a lot with what we have already.

When it is necessary to expand the system, I think it's best to try to make the new capabilities as much inline with the existing system as possible, so that they are relatively intuitive to the people using them. This is something I like about Alessandro's += operator idea. It's the sort of thing a developer or advanced user would immediately understand simply from seeing the existence of it in the documentation, without even needing to read the accompanying explanatory text.

[bxparks]

So I discovered through hours of trial and error that the--build-properties flag has a broken parser, so it cannot be a substitute for -D (arduino-cli version: 0.12.0.-rc3, on Ubuntu Linux 20.04.):

1. It seems to currently handle only -DMACRO or -DMACRO=2, but not strings like -DMACRO="This Is A String":

$ arduino-cli  compile --fqbn arduino:avr:nano:cpu=atmega328old  --build-properties 'compiler.cpp.extra_flags=-DPIN=2 -DSSID="This is a String"' PrintStringTest.ino
Error: invalid argument "compiler.cpp.extra_flags=-DPIN=2 -DSSID=\"This is a String\"" for "--build-properties" flag: parse error on line 1, column 40: bare " in non-quoted-field

2. It does not parse embedded commas properly:

$ arduino-cli  compile --fqbn arduino:avr:nano:cpu=atmega328old  --build-properties 'compiler.cpp.extra_flags=-DPIN=2 -DSSID="Embedded,Comma"' PrintStringTest.ino
Error during build: Error reading from slice (index:1): Invalid line format, should be 'key=value'

3. No amount of backslash escaping works:

$ arduino-cli  compile --fqbn arduino:avr:nano:cpu=atmega328old  --build-properties 'compiler.cpp.extra_flags=-DPIN=2 -DSSID=\"Embedded,Comma\"' PrintStringTest.ino
Error: invalid argument "compiler.cpp.extra_flags=-DPIN=2 -DSSID=\\\"Embedded,Comma\\\"" for "--build-properties" flag: parse error on line 1, column 41: bare " in non-quoted-field

4. Note that the Arduino IDE's --pref flag handles the parsing correctly, so the CLI's --build-properties is not a replacement for the IDE's --pref:

$ /home/brian/dev/arduino-1.8.13/arduino  --verify --board arduino:avr:nano:cpu=atmega328old  --pref 'compiler.cpp.extra_flags=-DPIN=2 -DSSID="Embedded,Comma"' PrintStringTest.ino
Loading configuration...
Initializing packages...
Preparing boards...
Verifying...
[...]
Sketch uses 6136 bytes (19%) of program storage space. Maximum is 30720 bytes.
Global variables use 717 bytes (35%) of dynamic memory, leaving 1331 bytes for local variables. Maximum is 2048 bytes.

5. With regards to the problem of defining SECRETS, you write:

A system for doing this has been established in the Arduino Web Editor:
https://arduino.github.io/arduino-cli/dev/sketch-specification/#secrets

Hmm, I've never heard of the Arduino Web Editor. I'm someone who writes code using the vim(1) editor and uses git(1) on the command line for source control. You want me to code in a web browser, while forced to be connected to the network? I'm not sure how this is the solution to the SECRETS problem.

6. With regards to the needing the -D flag to target different hardware, you write:

A macro is defined for each board according to ARDUINO_{build.board}:
https://arduino.github.io/arduino-cli/dev/platform-specification/#boardstxt

I think you misunderstood. I am not just looking for detecting ARDUINO_AVR_NANO or ESP8266. I have built various hardware devices, using different microcontrollers, each with different switches, buttons, sensors, displays on different GPIO pins (e.g. PIN=2 or PIN=D3) and so on. I want to compile the same program targeted to different hardware devices, with slightly different configurations using the -D flag in a shell script or a Makefile.

I also write a fair number of Libraries, which have test programs (sketches) which I need to run across many different microcontrollers to make sure that they run properly on various boards. It would be impractical to run multiple dozens of validation programs manually through the Arduino IDE, manually changing various configuration options, without being able to create scripts to automate the testing.

7. I am not in favor of overloading the --fqbn flag to hold the -D macros. Here is the fqbn for one of my ESP8266:

esp8266:esp8266:generic:xtal=80,vt=flash,exception=disabled,ResetMethod=ck,CrystalFreq=26,FlashFreq=40,FlashMode=dout,eesz=512K,led=2,ip=lm2f,dbg=Disabled,lvl=None____,wipe=none,baud=115200

Hiding a -D macro in there seems to be unnecessarily punitive to developers.

Additional Thoughts on the -D flag

I have to say that I am constantly surprised at the push-back against the -D flag. It is a feature that has been requested many times, by many people, over many years, as indicated by the original post. It's a significant pain point for many people. It's a simple, focused request. It is a well-defined feature, since it is directly supported by the gcc compiler, and is familiar to any programmer who knows about the gcc compiler. We don't even have to write a manual for it, we can just point people to the gcc compiler preprocessor options.

I am reaching a point where the -D feature might be the thing that finally breaks the camel's back. In other words, I am considering giving up on the Arduino platform without the -D flag. I spent 8 hours yesterday trying to get --build-properties flag to work with strings. It doesn't work, at least I couldn't find any way to escape the broken parser. I'm tired of fighting the Arduino tools. I would rather spend my time writing interesting Libraries that I can share with others, and write interesting Arduino applications.

[per1234]

So I discovered through hours of trial and error that the--build-properties flag has a broken parser

Thanks for reporting this. Would you mind opening a dedicated issue report? (UPDATE: now reported at #1037)

You want me to code in a web browser, while forced to be connected to the network?

Not at all. But note that link also explains how the system is applied to local sketches as well. I already explained how you could use that to solve your issue of needing to define secrets in a way that is not checked in to version control.

However, the main reason I mention it is because we always need to consider the systems that have been previously established in the other official Arduino development tools in order to provide as consistent a user experience as possible.

I want to compile the same program targeted to different hardware devices, with slightly different configurations using the -D flag in a shell script or a Makefile.

I understand now. Thanks for the explanation.

I have to say that I am constantly surprised at the push-back against the -D flag. It is a feature that has been requested many times, by many people, over many years, as indicated by the original post. It's a significant pain point for many people. It's a simple, focused request.

OK, well this is my proposal for providing that feature.

You didn't answer my question from before: Assuming it became possible to add quotes to property values set via --build-flags, would the proposal I present here then be sufficient to make you feel that Arduino CLI provides the "easy -D flag mechanism" you are requesting?

I spent 8 hours yesterday trying to get --build-properties flag to work with strings. It doesn't work, at least I couldn't find any way to escape the broken parser.

I'm sorry to hear that you had a bad experience using Arduino CLI. I hope that anyone who writes code will understand that bugs do happen sometimes. This project is a work in progress, hopefully steadily improving overall. High quality bug reports and feature requests are valuable contributions to that effort.

[matthijskooijman]

One addition here: If we provide such a set of "user-provided flags", we should probably make sure to provide both recipe-specific flags (e.g. only for .c files, or .cpp files, or only for linking, maybe even also for the ar and elf2hex recipes) and a property to add common flags (passed to the c, cpp and S recipes, maybe also to the link, though that's probably not required). That should make it convenient to just add extra flags "to the build", but also allow full control for flags that apply to only one or two filetypes.

One flag var that is missing in the first post is the one for recipe.preproc.macros, which should probably also get a user-changeable flags variable.

Furthermore, I would suggest using the name link (rather than c.elf) for the link flags (e.g. build.user_link_extra_flags) to make it more clear where it is used (an extension of that would be to rename the recipe itself, too, but that's a different issue).

What would be the next step to move this forward? I think writing a more detailed proposal about which properties a core is recommended to expose (ideally in a form that can be included in the documentation as-is)? This proposal would at least need to talk about the "user-facing" properties, but maybe it could also make some (more casual) recommendations about internal properties to use (for consistency)?

[bxparks]

Broken Parser in --build-properties

It seems to me that the--build-properties flag, as currently defined, cannot provide the functionality of the -D flag because the comma-character is used as a delimiter: --build-properties strings List of custom build properties separated by commas. Or can be used multiple times for multiple properties.

Let's say that we fix the current parser so that the comma-character is ignored inside a double-quote. That doesn't fix the parsing, because the following will not work:

#define COMMA ,
#define GREETINGS {"hello", "hey", "g'day mate"}

Maybe we attempt to fix that by requiring the right-side of compiler.cpp.extra_flags= to be encoded in a single quote, like this: --build-properties "compiler.cpp.extra_flags='-DPIN=2 -DSSID=\"This is a String\"'". But we can break that parser trivially by attempting to encode this:

#define NEWLINE '\n'

Any attempt to add backslash-escaping of the comma (\,), the single-quote (\'), or the double-quote (\") will result in an incomprehensible nesting of multiple backslashes, as we try to escape the quoting and backslash mechanism of the bash shell.

The only way I can see --build-properties working is to allow only a single xxx=yyy Java .properties line to be associated with a single instance of the --build-properties flag.

Unreasonable Complexity of --build-properties

My biggest issue with the --build-properties is the unnecessary complexity of this solution for solving such a simple problem. Application and Library developers want a simple thing, the -D flag. They should not need to know the details of how Arduino platforms are assembled and configured. In particular, application and library writers should not need to know the internal defails of the various boards.txt and platform.txt files.

The original propose in this ticket, with regards to splitting the extra flags between "platform" flags and "end-user" flags solves the problem with accidental clobbering. But I don't think it solves the problem of unnecessary complexity. I don't think the Application and Library writer should need to know that the configuration they need to override is in the boards.txt file, and it is called compiler.cpp.extra_flags (or under @per1234 proposal, it might be compiler.cpp.user_extra_flags.user).

I have a similar feeling about the += syntax. I don't think end-users (i.e. app and library developers) should need to go into the internal details of boards.txt.

Thoughts on arduino_secrets.h

I've never used the Arduino Web Editor. If I understand the https://arduino.github.io/arduino-cli/dev/sketch-specification/#secrets page correctly, the arduino_secrets.h file is generated upon downloading of the sketch. I have a couple of thoughts:

1. I personally don't make much operational distinction between "secrets" (such as SSIDs and PASSWORDs) and configurations such as:

• #define LED_PIN 3

• #define BUTTON_PINS {4,5}

  Both types of things are configuration parameters that need to be injected into an app using a -D flag. I agree that there is some semantic difference in how carefully we should treat SECRETs. But for the purposes of Arduino programs, I think they can be treated about the same. After all, not many people will need to use encrypted KMS (Key Management Systems) for Arduino secrets. And if they did, they could still use the -D flag to inject those secrets.

2. There seems to be an assumption that a single .ino file has only a single set of secrets or a single set of configurations. For many people asking for the -D, I don't think this is the case. For example, I have 2 SSIDs, an "internal" SSID and an "external" SSID. Certain IoT devices go on certain networks. (I actually have 6 SSIDs, but I'm simplifying here.) Sometimes I want to place the same Arduino app on different networks, on different hardware devices, configured slightly differently, for debugging purposes. Having a single arduino_secrets.h file will be limiting. I would probably write a script to dynamically generate the arduino_secrets.h file. But if I'm going to do that, it would be far easier to just directly call the compiler with the appropriate -D flags.

   It looks like I currently maintain about 200 .ino files. It is impractical to maintain 200 arduino_secrets.h files. What I have is a set of 5-10 hardware devices (with known PIN numbers and other I/O configurations), a set of 6 SSIDs, and some other secrets, all of them organized into about 10-15 "environments". The 200 INO files are frequently validated against the 10-15 environments, not all of them on all environments, but maybe the total of 400-600 different combinations of INO files and environments. The only way to maintain these is through automated scripting, and validating them through continuous integration.

Summary

1. I think the current --build-properties flag is broken. I think it can be fixed by removing the support for comma-delimiting.
2. I think the --build-properties is an unnecessarily complex solution to the simple problem of -D. Splitting the various properties into 'platform developers' versus 'end-users' does not solve the complexity problem.
3. The arduino_secrets.h is not a good substitute for -D flag when there is a combinatorial explosion of INO programs, combined with various secrets and configuration parameters.
4. I think the -D is the simplest solution, that easily digestible by "end-users" (app and libraries developers), that solves the widest range of configuration problems (secrets and configurations).

Given how long and how often the -D flag has been requested, I am sure that I am not making any new arguments. So I expect my request to be rejected as well. But if a miracle happens and the proposal is accepted, then I can also volunteer to do the work. I will learn the Go Language just to implement this feature. (I'm already pretty fluent in C, C++, Java, Python and JavaScript. Go should not be impossible.)

[per1234]

I think the current --build-properties flag is broken. I think it can be fixed by removing the support for comma-delimiting.

If I understand #532 correctly, it's a report of this same type of problem caused by the support for a comma-separated list of properties, and also proposes the comma-separated list support be dropped.

[matthijskooijman]

If I understand #532 correctly, it's a report of this same type of problem caused by the support for a comma-separated list of properties, and also proposes the comma-separated list support be dropped.

Yup, that seems to be exactly this problem, so let's continue discussion about it there, and assume it will be solved somehow.

In particular, application and library writers should not need to know the internal defails of the various boards.txt and platform.txt files.

The purpose of this issue is to define a standard set of properties, so that users do not need to know the internals of the platform.txt file. If this standard property set is adopted, it can be documented in the arduino-cli manpage, and users do not need to look further than that.

The only caveat is that existing cores will not automatically and instantly adopt the new properties, so it might be important to let cores that support the new properties define them with empty values in platform.txt, as an indication that they support it. If so, arduino-cli could maybe provide a warning when setting a property that is not defined (with an empty value, maybe only specifically for these properties?).

However, since platform.txt is the canonical source for how to run the compiler, it has to be involved when adding compiler -D options somehow, and I think there is currently no standardized attribute that the IDE/arduino-cli can define that is always included in the build. There is {includes}, which could work, but that would be an abuse. arduino-cli could also just append something to the resulting recipe, but that's a hack that will bite us in the future, so I'm strongly against that.

Hence, the only reasonable way to add flags to the commandline, is by defining a standardized set of build properties like this issue suggests.

What we could maybe do, of course, is in addition to the --build-properties flag add something like --extra-define FOO=BAR, which is a convenience helper that internally just does the same as --build-properties build.user_extra_flags=-DFOO=BAR, so it uses the same interface to the platform, but improves the interface for the user.

One additional suggestion: maybe it would be good to define three sets of properties. The two suggested before (one for the platform/board developers to set internally, one for the user to set through e.g. --build-properties), and then a third to be set by the tooling (i.e. arduino-cli). This keeps the door open for arduino-cli to generate additional flags when needed later (I can imagine at some point we allow libraries to declare compile-time options in some generic format, converted into -D options by arduino-cli, or the --extra-define suggested above could also use this "tooling" property set, to prevent conflict with manually supplied --build-properties).

[matthijskooijman]

Since sketches.ino are parsed first, is it a silly idea to hijack gcc's #pragma directive like this:

I think that having some way to configure build flags from a sketch would be a useful addition, but I do not think that this is the right issue to dicuss this. I would want to keep this issue focused on standardizing the build properties offered by cores to allow customizing the build flags in a generic way. If we solve this issue, we can build a per-sketch-flags feature on top of it. As for the per-sketch-flags feature, I know this has been discussed repeatedly in the past, but I'm not sure if there is any currently open issue to track this? Closest I could find is arduino/Arduino#421, but that's really an old and broader issue. @per1234, do you know a better one to refer to?

Coming back to the topic of this issue, I previously said:

One additional suggestion: maybe it would be good to define three sets of properties. The two suggested before (one for the platform/board developers to set internally, one for the user to set through e.g. --build-properties), and then a third to be set by the tooling (i.e. arduino-cli). This keeps the door open for arduino-cli to generate additional flags when needed later (I can imagine at some point we allow libraries to declare compile-time options in some generic format, converted into -D options by arduino-cli, or the --extra-define suggested above could also use this "tooling" property set, to prevent conflict with manually supplied --build-properties).

If some way is added to define per-sketch flags, I think these could be included in the "tooling" set as well, so no extra changes are needed here to prepare for per-sketch flags later.

[egnor]

In theory this proposal is basically a documentation change, along with a lot of elbow grease and cat-herding to get the proposal (e.g. build.user_defines) implemented, right? It doesn't need major (or any) changes to the Arduino IDE or CLI code base that I can see?

And it's necessary for any effort to allow project-specific definitions (e.g. arduino/tooling-rfcs#9 where I recently spelled out many use cases and why standard workarounds aren't great).

The one thing I don't know how to readily manage is that different compilers may have different syntax for specifying defines (-DFOO=BAR vs /DFOO=BAR, different conventions for quoting arguments, etc). I'm not sure how easy it is for board/platform definitions to translate from something common into whatever their compiler expects. Maybe this is already an issue platform definitions have to deal with?

[per1234]

In theory this proposal is basically a documentation change

That is correct.

a lot of elbow grease and cat-herding to get the proposal (e.g. build.user_defines) implemented

The necessary changes to implement these properties in each platform are quite minimal, straightforward, and risk-free. The undocumented compiler.*.extra_flags convention is already implemented in many platforms.

It doesn't need major (or any) changes to the Arduino IDE or CLI code base that I can see?

None at all. It leverages the standard platform configuration functionality that is already in place in any Arduino development tool in use today (even in the outdated 1.x versions of Arduino IDE).

The one thing I don't know how to readily manage is that different compilers may have different syntax for specifying defines (-DFOO=BAR vs /DFOO=BAR

It is not relevant to this proposal. The user passes the complete arguments via the property so the user simply must use the appropriate syntax.

[egnor]

[Syntax for defines] is not relevant to this proposal. The user passes the complete arguments via the property so the user simply must use the appropriate syntax.

So the user would need to know the syntax of the compiler in use by the platform? (And how to quote values, etc.?)

[per1234]

Yes. These properties are intended to allow advanced users to do advanced things. The user must understand the compiler's command line interface before they start injecting arguments into the commands.

Non-advanced users don't have any need for this functionality. If a developer is designing their project in a way that forces a novice user to use this feature, that is a mistake on their part and we will not make any effort to cater to such misuse of the feature.