Feature: Minimum/maximum compatibility mode

[matthijskooijman]

This is something that I've been thinking about for quite some time. I might have proposed it already sometime, but I couldn't find anything :-) This is not strictly something for arduino-cli, but more something for the Arduino ecosystem as a whole, but this seemed like a good place to at least discuss this.

One of the things I've seen in the past is that making change to the Arduino cores to add features or fix bugs is hard when they affect compatibility. There are quite some things where we would want to fix things or change defaults, but breaking existing libraries prevents us.

Examples are setting a default timeout on the Wire library, a dummy Print::flush() implementation, the -fpermissive flag, deprecated functions in the SPI library and a recent one that triggered me writing this, is making failed memory allocation terminate instead of returning NULL.

In a lot of cases, one can create a deprecation scheme for this compatibility, e.g. by 1) first introducing an alternative function or extra argument, then 2) encourage people to switch to this new function or explicitly pass this argument, and then later 3) remove the old function or switch the default for the argument. If done properly, this produces sketches and libraries that are backward compatible across step 3 (e.g. a sketch will work with any core version after step 1), sometimes even across step 1 too.

However, encouring people to switch to a new function or similar is currently quite hard. We can update documentation, but that doesn't reach the right audience usually. Ideally, there would be some way where people could be automatically notified of potential incompatibilities in their code, helping them to future-proof their code.

What I envision we could implement, is three modes of compilation:

1. The "Minimum compatibility" mode disables / removes everything that is deprecated or discouraged, causing any code that uses it to compile. This mode can be used to check your code and helps making it future-proof. We should encourage advanced users and library authors to use this mode by default.
2. The default mode is pretty much as we have now, except we could maybe add some deprecated attributes to produce warnings.
3. The "maximum compatibility" mode re-enables (some) constructs that have been removed from the default mode already. This can be used when you need to compile an older sketch or library that has not been updated for a long time.

When something is removed and/or changed in a backward incompatible way, this changes is first made only for mode 1. After some time, when people have had opportunity to use mode 1 to future-proof their code, the changes are also mode for mode 2. If feasible, the change should never be made for mode 3 (i.e. in mode 3 old stuff should be supported indefinitely), but I suspect that in practice this would prevent further improvements, so things might also disappear from mode 3 as well (alternatively some new features would not be enabled in mode 3).

One question is how should the user configure this mode? I see two options:

1. The obvious way would be to add a configuration directive / UI in the tooling (arduino-cli, Arduino IDE), similar to how verbose building and warnings are implemented. Actually putting the -D option on the commandline should probably involve platform.txt somehow (like with warnings, or the -DARDUINO option).
2. An alternative option would be to let each compare handle this separately by using menus (i.e. duplicating the menu for each board, or platform-wide if Feature Request: Support platform-wide "menu" options #922 is implemented).

Even though option 2. would be simpler (no changes to be coordinated and duplicated among all tooling, other than implementing #922), I do think 1. might prove more powerful. It would allow a user to configure this globally (unlike platform options which would probably be reset whenever you switch boards or platforms) and give the option a more official status (instead of each platform maybe defining slightly different meaning to each mode, or offering different modes). It would also mean that libraries could also respond to the same macros in addition to just the cores, making things even more powerful (this does require that we globally specify how the macro(s) is/are named and what their meaning is exactly).

One important aspect of this is of course documentation. Having a technical mechanism to check for compatibility issues is one thing, but we should make sure that cores and libraries have a good list of all stuff that is different between each of these modes, ideally with some hints on how to migrate existing code.

[paschlie]

I'd vote for a general cleanup and thoughtful commenting of the all the Arduino library headers and sources in general, and require at least their root #inclusion in a project source file be explicit, and preceded by a specific #defines if certain depreciated capabilities/behaviors are required by the source; thereby enabling incompatible revisions/refinements to be made, but allowing depreciated behaviors to be at least temporarily invoked upon demand for some period of time. (As incompatible refinements are more likely the exception to the rule than not, this should not represent an unreasonable burden to the vast majority of users, I believe.)

With respect to the memory allocator terminating execution upon failure, I actually believe this is a mistake, believing its better to enable the users code to handle the exception signaled by a return of null; and which can easily be wrapped in another function call to enable your preference as may be desired.

[matthijskooijman]

I'd vote for a general cleanup and thoughtful commenting of the all the Arduino library headers and sources in general, and require at least their root #inclusion in a project source file be explicit,

That would be useful, but heavily break compatibility and possibly also conflict with the ease-of-use for novice users that Arduino aims for.

and preceded by a specific #defines if certain depreciated capabilities/behaviors are required by the source;

This is tricky: By preceding an include by a define, you can influence whether a specific function or method is declared, but its definition will be in a separate .cpp/.c file that will not have the define, so will still always be included (which can work for functions, which are simply not linked, but not for methods, which produce incompatible classes in different source files).

The only way to really ensure that such "compatibility defines" are properly accounted for, is to pass them as -D options on the compiler commandline, so they are defined for the entire source. Hence my suggestion to do this with a global compatibility mode.

[paschlie]

I'd vote for a general cleanup and thoughtful commenting of the all the Arduino library headers and sources in general, and require at least their root #inclusion in a project source file be explicit,

That would be useful, but heavily break compatibility and possibly also conflict with the ease-of-use for novice users that Arduino aims for.

I don't see how cleaning up and better documenting the header and implementation files would break anything.

and preceded by a specific #defines if certain depreciated capabilities/behaviors are required by the source;

This is tricky: By preceding an include by a define, you can influence whether a specific function or method is declared, but its definition will be in a separate .cpp/.c file that will not have the define, so will still always be included (which can work for functions, which are simply not linked, but not for methods, which produce incompatible classes in different source files).

The only way to really ensure that such "compatibility defines" are properly accounted for, is to pass them as -D options on the compiler commandline, so they are defined for the entire source. Hence my suggestion to do this with a global compatibility mode.

I don't believe Arduino API refinements typically necessitates breaking backward compatibility; and if there is a source-level API change, it's certainly not a complier issue.

---

(In general, I actually believe the current IDE does a disservice to beginners by attempting to hide the truthfully critical inclusion of header files in code sources, as it is these very headers which document the API and thereby the interpretation of the users code by the compiler. Further, not enabling the convenient review of of such headers actually hurts beginners, as encouraging their inspection will actually greatly help beginners understand what's going on when they get a compiler error regarding an Arduino environment supported function call, not hurt.)

[matthijskooijman]

I don't see how cleaning up and better documenting the header and implementation files would break anything.

You said:

and require at least their root #inclusion in a project source file be explicit,

Which I interpreted as "If you want to use Serial, you must explicitly include e.g. <Serial.h>". That would break compatibility, since existing sketches do not have such includes. Maybe I misinterpreted your intention.

Cleanup and better separation of headers is a good idea, but I think is already done in part in the new https://github.com/arduino/ArduinoCore-API repository (and I think that any further discussion about such cleanup is better placed there than in this issue).

[paschlie]

I don't believe it's too much to require that something like "#include <arduino.h>" precede user code, just as users are required to define "setup()" and "loop()" function calls, and which can be easily included in a new project's template file as the required two function templates are today. This header file can by default contain other #includes as may be required to support the whole Arduino base API, and other augmented API libraries as may be desired, and who's inspection should be encouraged, not inhibited as they are now.

[matthijskooijman]

I don't believe it's too much to require that something like "#include <arduino.h>" precede user code, j

Whether or not Arduino.h should be included or not is, I think, a discussion completely separate from this one (there might even be an issue about this already somewhere). I actually agree with you here, if we would not have compatibility to take into account, I would gladly and enthusiastically drop the implicit Arduino.h include and have people include it explicitly. But I think that's a separate discussion (though if the "Min compatibility mode" is implemented as I suggest, then not automatically including Arduino.h could be one of the things that it results in).

[paschlie]

I don't believe it's too much to require that something like "#include <arduino.h>" precede user code, j

Whether or not Arduino.h should be included or not is, I think, a discussion completely separate from this one (there might even be an issue about this already somewhere). I actually agree with you here, if we would not have compatibility to take into account, I would gladly and enthusiastically drop the implicit Arduino.h include and have people include it explicitly. But I think that's a separate discussion (though if the "Min compatibility mode" is implemented as I suggest, then not automatically including Arduino.h could be one of the things that it results in).

What incompatible API changes are your suggesting, or is your argument hypothetical?

In general I believe API level changes, as should be reflected in source headers should be managed by facilities already available to them for this purpose; and thereby don't need more magic behind the scene which more likely than not, only in-effect unnecessarily obfuscates the users ability to understand what's going on. Again header files serve a very specific and necessary purpose, which especially beginners need to understand early in the process of learning to program, as they in combination with the language being used, define the API available to them. I for one believe.

[matthijskooijman]

What incompatible API changes are your suggesting, or is your argument hypothetical?

In this particular quote, I would suggest to stop automatically including "Arduino.h". This is incompatible, because existing sketches do not include it and would break (it's trivial to fix, but it's still incompatible and would be non-obvious for a lot of users).

In the general sense, every now and then situations come up where APIs could be improved or simplified if compatibility could be (gradually, e.g. through deprecations) broken. I've included some examples in the initial post, with a description of how this could work.

As for solving this all in headers, I do not think this is possible in the general sense. Take the example of changing the behaviour of allocation failures. In my sketch I could write:

#define MINIMUM_COMPATIBILITY_MODE
#include <Arduino.h>

which could tell the relevant header file that I want minimum compatibility and thus the new allocation failure behaviour. However:

• Other source (.cpp/.c) files that also include Arduino.h will not define this, so they will get the header file unmodified (unless you modify all source files in your sketch and all libraries).
• The implementation of this behaviour is in cores/arduino/new.cpp, which also does not see your define, so has no way to know it should implement the new behavior.

To solve changes like these, or implement opt-in more strict checks or warnings for deprecated stuff, I really cannot see any other way than to use -D options on the commandline for all source files, which needs to involve the tooling somehow and cannot be implemented in the headers.

Again, maybe I am totally misunderstanding your intent here, but I have the impression that you might be misunderstanding my proposal and our discussion might only be distracting from the original topic...

[paschlie]

To begin:

• change the default new project stationary to include "#include <arduino.h>", which need not be equivalent to "Arduino.h", although will likely include it among other base library headers, defining the base Arduinio API.
• while initially continuing to support their magical inclusion behind the scenes, and thereby not breaking anything for existing projects, begining the transition to a requirement for their explicit top-level inclusion.
• add a feature to the IDE to make it easy to open and view included baseline headers, and and those from optionally supported libraries (but possibly kept read-only by default, so beginners don't mistakenly modify them).

[matthijskooijman]

That sounds like a sane approach and I like it, but is just one particular improvement that is really orthogonal to the improvement suggested in this issue. If you are serious about your suggestion, you should create a new issue about it so it can be separately discussed.

[paschlie]

Agreed, however I don’t believe it’s strictly orthogonal to your suggestions; as it bootstraps a means by which library API’s can be enabled to evolve though time, in an orderly and visible manner, just like all library API's have historically. (With respect to the linker, once library inclusion has been made more explicit, the source file may be parsed by the tool to determine what library dependencies may exist, and direct them to be linked in the final build). Best regards.

…

On Sep 18, 2020, at 11:55 AM, Matthijs Kooijman ***@***.***> wrote: That sounds like a sane approach and I like it, but is just one particular improvement that is really orthogonal to the improvement suggested in this issue. If you are serious about your suggestion, you should create a new issue about it so it can be separately discussed. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#956 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOQAXGRI54NMHPOUCKSI6TTSGN7ITANCNFSM4RQPJO7Q>.