Letting Arduino/arduino-cli compile unittests

[matthijskooijman]

While thinking about #146 (compiler flags), I realized that the regular Arduino builds potentially use different flags for C and C++ files. Then there is #113 (compile object files separately), which would require complicating the build process further. Then I realized that some libraries are intended to be compiled with "dot_a_linkage", which we might be able to ignore since it is rarely used, but still.

Anyway, the bottom line is that for compiling unittests, you would ideally mimic the regular Arduino build process as close as possible. Then I realized: Maybe Arduino can actually just do the build, using regular gcc, producing a binary? This could be done by making an Arduino Platform/core that just contains the mocked code, and then each unittest would just be a separate sketch.

Looking around to see if there is any existing such core, I found only ncore which has not been updated since 2012 and is still written for Arduino 1.0, so probably not a good starting point.

With this approach, the current arduino_ci code could be split: One part would be an Arduino core, along with a unit-test library, that could be installed as a normal Arduino core and also used from the Arduino IDE directly (where maybe "upload" would just run the generated program, showing output in the IDE console). Then the second part would be the sketch-compile-test and unit-test-compile-and-run driver, that uses Arduino to do the grunt work.

A related observation is that using the Java IDE to compile things, as is done now, is a bit heavy. A good alternative would be to use arduino-cli, maybe you already saw that? It's a somewhat recently introduce commandline tool that can do pretty much everything the IDE can (building sketches, instaling libraries, cores, getting info about all kinds of things), but with a proper CLI interface. I think it's currently used as the backend for the new (alpha) Arduino Pro IDE, and is intended to become the backend for the Java IDE as well at some point.

All this would be quite an invasive change, that puts arduino_ci pretty much upside down, but at first glance it does look like an interesting direction...

[matthijskooijman]

Note that with arduino-cli, you can still customize the build (e.g add flags or defines) by passing --build-properties to arduino-cli compile (provided that the core has a flags build property that can be overridden, which we could arrange in our custom core, and for other cores this will be standardized in the future too: arduino/arduino-cli#846).

[ianfixes]

There are a few architectural things that don't quite make sense here.

1. Arduino uses avr-g++. I'm not sure what level of hacking would be required to get it to use something different with different flags
2. Arduino builds executables that wrap the definitions of setup() and loop() in user code, as the hardware doesn't really have the concept of a process that starts and terminates
3. Its not clear to me how Arduino would turn Serial.write (for logging messages from an uploaded sketch) into STDOUT when running on x86.

There is a dummy implementation of using the arduino CLI in https://github.com/ianfixes/arduino_ci/blob/master/lib/arduino_ci/arduino_cmd_linux_builder.rb

This project predates that one by a little bit but it was something of interest back then that could probably be revisited.

[matthijskooijman]

Arduino uses avr-g++. I'm not sure what level of hacking would be required to get it to use something different with different flags

This is no longer hardcoded sinds Arduino 1.5, since that supports also non-AVR targets. All the compiler commandlines are no pretty much entirely configurable, only the structure (the order in which commands are run on which files) is hardcoded. Configuration happens in the platform.txt file for a platform, see e.g. https://github.com/Arduino/ArduinoCore-samd/blob/master/platform.txt for the SAMD version, and https://arduino.github.io/arduino-cli/latest/platform-specification/ for the specification of this file and custom cores in general.

Arduino builds executables that wrap the definitions of setup() and loop() in user code, as the hardware doesn't really have the concept of a process that starts and terminates

Correct, but this is only so because the particular Arduino core has a main() function that calls these functions: https://github.com/arduino/ArduinoCore-samd/blob/356a11aae7229669cf864f58bcaf0fb0e1906123/cores/arduino/main.cpp#L33. A "unit testing" core could have a main function that does something else entirely (though I also like the idea of being able to run arbitrary Arduino sketches on a PC without modifications, so support for setup/loop might be a good optional feature still).

Its not clear to me how Arduino would turn Serial.write (for logging messages from an uploaded sketch) into STDOUT when running on x86.

Just like your current code would do this: Just let the core define a custom Serial object that implements write in that way. A custom core is in full control, it can (and has to) specify all code that will be compiled along with the sketch.

This project predates that one by a little bit but it was something of interest back then that could probably be revisited.

Yeah, arduino-cli is still fairly new and under heavy development, but I think it is already way easier and more complete than the Java IDE commandline is (and a lot better designed, the Java IDE commandline was a bit of hack in some places).

[ianfixes]

This sounds promising and I have no real objection to it. I'm still a bit nervous about hard-coding the compiler stuff into platform.txt, if only because I built a feature around dynamically testing what the compiler supports. That allowed me to find problems like this: adafruit/Adafruit-WS2801-Library#20

(see the build log under AddressSanitizer: heap-use-after-free)

If that can be sorted out (even in a hacky way). I would absolutely support a contribution like this, since it would allow me to drop a lot of the bundled Arduino C++ libraries from this codebase.

Is there a proof of concept you can show of the file set (and short shell script) for creating and running a non-looping x86 binary?

[matthijskooijman]

Hey @ianfixes, I generated a bit of traffic on your issue queue with a bunch of sugestions last week. However, just a quick headsup that I'm not planning on working on these things just now. I realized that all this was mostly a distraction from the actual work I needed to be doing, and while it would be helpful if all I suggested were already finished, I can't warrant the time it would take to build it just now. So I'm going to keep going with our very basic, but existing arduino dummy code and hack some things in there, at least until I've made some progress in the rest of my project. I still hope to revisit this (or at least the subject) later, though!

[ianfixes]

There is no hurry, this is an unfunded Apache-licensed project built by volunteer efforts -- in the complete absence of any official endorsement from the Arduino team. I respond to incoming suggestions to sort out what is (and isn't) compatible with the project's design goals or fundamental assumptions, but I'm not producing much code during the pandemic.

So don't worry about whether you can contribute code; your ideas are good and worth exploring in the future. I'm glad you shared them here.

[ianfixes]

@matthijskooijman I think this work will depend on further ardunio-cli support. And I believe that since you opened this, I've adopted ardunio-cli as a backend -- I'm effectively "all in" on that.

My current thinking here is that I will need to properly mock avr-libc for avr boards, and other builtin libraries as appropriate for those other architectures. If I use a single "x86 core" to accomplish this, I would lose the value of running unit tests against different board platforms (which is intended to catch things that are hidden behind preprocessor directives).

But to accomplish that, I'll need to be better aligned with the "official" compilation of those libraries. So I think this issue now depends on these feature requests upstream:

• Surface compiler flags for a given platform arduino/arduino-cli#1090
• Feature request: Expose sets of library files and directories needed for the compiler arduino/arduino-cli#1092

Does it sound like I'm going about this the wrong way? Again, ultimately I have to be able to run the compiled unit test binaries without microcontrollers present -- on the x86 host itself. That rules out doing the compilation 100% on the native compiler (e.g. avr-gcc).