Enable unit tests - ignore a test/ directory in an Arduino library

[ianfixes]

Background

I've developed a system called arduino_ci that enables Arduino libraries to be unit tested, both locally and remotely (as part of a service like Travis CI). Among other things, this includes abilities such as

• Fully simulating the physical world (e.g. clock millis(), pin inputs like digitalRead(), serial inputs, etc)
• Testing against multiple boards with no hardware present
• TAP-formatted test output

Problem

The trick is where (in the library) to put the files that contain the tests. According to the specification:

An extras folder can be used by the developer to put documentation or other items to be bundled with the library. [...] The content of the extras folder is totally ignored by the IDE; you are free to put anything inside such as supporting documentation, etc.

I would prefer not to put tests in "extras/", both for closer proximity to the code begin tested and because (in my opinion) the ability to run tests is an important and necessary part of any software project. (Similar to how they are organized in modules for Java, Python, NodeJS, etc.)

Question

What chance is there of amending the library specification to ignore (whether that means whitelisting or blacklisting in this context) a directory called "test/" where all the automated tests would live? Alternately, could something similar to .gitignore be used for that purpose?

See Also:

• Initial Arduino forum thread
• Arduino forum thread that led me here
• StackExchange writeup with more information
• Example of arduino_ci unit testing the Queue library
• Example of arduino_ci unit testing the Adafruit FONA library

[facchinm]

Hi @ianfixes ,
the project is super nice! About the test folder, can you expand on that? Do you expect files under test to appear in the menu (as they would if the extension was .ino)?
Or do you simply would like to introduce the test folder as a standard container for tests?

[ianfixes]

I don't require any UI interaction. I think I'm almost better off if the IDE completely ignores the contents of test/ because I don't think the capability to test an Arduino library should be associated with my system in particular. I just feel that there should be a standardized place to keep unit tests (nearer the code than a subfolder of extras/) and the user is free to choose from any number of Arduino unit test paradigms, to find one that works for them.

That said, if you do want to consider using arduino_ci as a standard method for unit test / CI support, I would recommend bundling g++ with the Arduino IDE install (not avr-gcc, but something that can actually build on the host architecture -- where the tests run).

But the answer to your question is (TL;DR) just introduce test/ as a standard container.

[per1234]

I think what is being requested is for some text to be added to the library specification:
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#layout-of-folders-and-files
Something like:

Tests

A tests folder can be used by the developer to store test files to be bundled with the library.

The content of the tests folder is totally ignored by the IDE.

Then ensure that the Arduino IDE complies with the specification by ignoring that folder.

As far as I know, it will almost do so already. The only exception is that any sketches under the tests folder will be listed under the File > Examples > {library name} > tests. Even without specifically incorporating the tests folder into the specification, that behavior is not what I'd expect from reading the specification, which seems to suggest that only sketches under the examples folder should appear in the File > Examples menu:
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#library-examples

Library examples must be placed in the examples folder.

Sketches contained inside the examples folder will be shown in the Examples menu of the IDE.

In fact that behavior already violates the specification when sketches are placed under the extras folder (arduino/Arduino#5688) so that needs to be changed anyway.

[ianfixes]

Yes, this suggested text looks perfect!

I'm not sure if this would cause a related issue where the IDE would try to pull in test/*.cpp for compilation if someone included a library as a dependency.

[ianfixes]

As the arduino_ci GitHub Action takes shape, it becomes more difficult for me to change things later.

If there is any great objection to calling the directory test/ (note singular spelling, not plural), I'd value that feedback as early as you can provide it.

[ianfixes]

Opened #1125