For your consideration: Enabling unit tests on Travis CI using a different CI script

[ianfixes]

Purpose

Why	To enable Arduino library developers to accept contributions more easily, and increase the amount of GitHub collaboration
What	A cross-platform framework for compilation-verification and unit-testing Arduino libraries
How	By mocking out the Arduino hardware (and providing automation scripts that can run both locally and remotely), maintainers can enforce tests on code without requiring hardware present

Context: this repo doesn't provide for unit testing, and PlatformIO isn't Free Software. But install.sh inspired me to write my own alternative CI system called arduino_ci that accomplishes both, and it seems appropriate to submit my work for your consideration.

Live Demonstrations

Adafruit FONA library

This pull request against the FONA library shows arduino_ci as a drop-in replacement for travis-ci-arduino. It includes an example of mocking out a hardware device (in this case, a modem) to perform what would otherwise be a hardware-in-the-loop test.

In other words, the fake serial device allows us to see whether expected hardware responses produce the appropriate behavior in the library.

Adafruit WS2801 library

This pull request against the WS2801 library shows arduino_ci running in a parallel TravisCI job with travis-ci-arduino, to show that the run times are roughly equivalent.

Additionally, the unit test runner uses memory address sanitization to isolate memory access issues. This stack trace in an arduino_ci build job shows the source of a bug for which I've submitted the corresponding fix

Capability not present in travis-ci-arduino

• Unit testing capabilities give Travis CI the ability to validate the logic as part of the pull request process.
• Testing configuration is provided by a set of YAML files (all optional -- default behavior is identical to install.sh). This opens the possibility of parallelized testing by launching several different jobs with their own configuration.
• Ability to specify custom boards for download and mocking. This is of particular interest to manufacturers of boards :)

[ianfixes]

Brief update: as of v0.1.10, arduino_ci is now fully automated across Windows, OSX, and Linux hosts. You can write your tests under one OS and have them run and pass under another OS with the same outcomes.

Crucially, you can run the unit tests both locally and remotely with the same script.

[s-celles]

Any news? This is part of a more general issue : having better quality for Arduino libraries
arduino/Arduino#7567

[ladyada]

closing since we're not supporting travis anymore