Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable unit tests - ignore a test/ directory in an Arduino library #990

Open
ianfixes opened this issue Mar 13, 2018 · 6 comments · May be fixed by #1125
Open

Enable unit tests - ignore a test/ directory in an Arduino library #990

ianfixes opened this issue Mar 13, 2018 · 6 comments · May be fixed by #1125
Labels
type: enhancement Proposed improvement

Comments

@ianfixes
Copy link

Background

I've developed a system called arduino_ci that enables Arduino libraries to be unit tested, both locally and remotely (as part of a service like Travis CI). Among other things, this includes abilities such as

Problem

The trick is where (in the library) to put the files that contain the tests. According to the specification:

An extras folder can be used by the developer to put documentation or other items to be bundled with the library. [...] The content of the extras folder is totally ignored by the IDE; you are free to put anything inside such as supporting documentation, etc.

I would prefer not to put tests in "extras/", both for closer proximity to the code begin tested and because (in my opinion) the ability to run tests is an important and necessary part of any software project. (Similar to how they are organized in modules for Java, Python, NodeJS, etc.)

Question

What chance is there of amending the library specification to ignore (whether that means whitelisting or blacklisting in this context) a directory called "test/" where all the automated tests would live? Alternately, could something similar to .gitignore be used for that purpose?

See Also:

@facchinm
Copy link
Member

Hi @ianfixes ,
the project is super nice! About the test folder, can you expand on that? Do you expect files under test to appear in the menu (as they would if the extension was .ino)?
Or do you simply would like to introduce the test folder as a standard container for tests?

@ianfixes
Copy link
Author

ianfixes commented Apr 27, 2018

I don't require any UI interaction. I think I'm almost better off if the IDE completely ignores the contents of test/ because I don't think the capability to test an Arduino library should be associated with my system in particular. I just feel that there should be a standardized place to keep unit tests (nearer the code than a subfolder of extras/) and the user is free to choose from any number of Arduino unit test paradigms, to find one that works for them.

That said, if you do want to consider using arduino_ci as a standard method for unit test / CI support, I would recommend bundling g++ with the Arduino IDE install (not avr-gcc, but something that can actually build on the host architecture -- where the tests run).

But the answer to your question is (TL;DR) just introduce test/ as a standard container.

@per1234
Copy link
Contributor

per1234 commented Apr 27, 2018

I think what is being requested is for some text to be added to the library specification:
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#layout-of-folders-and-files
Something like:

Tests

A tests folder can be used by the developer to store test files to be bundled with the library.

The content of the tests folder is totally ignored by the IDE.

Then ensure that the Arduino IDE complies with the specification by ignoring that folder.

As far as I know, it will almost do so already. The only exception is that any sketches under the tests folder will be listed under the File > Examples > {library name} > tests. Even without specifically incorporating the tests folder into the specification, that behavior is not what I'd expect from reading the specification, which seems to suggest that only sketches under the examples folder should appear in the File > Examples menu:
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#library-examples

Library examples must be placed in the examples folder.

Sketches contained inside the examples folder will be shown in the Examples menu of the IDE.

In fact that behavior already violates the specification when sketches are placed under the extras folder (arduino/Arduino#5688) so that needs to be changed anyway.

@ianfixes
Copy link
Author

Yes, this suggested text looks perfect!

I'm not sure if this would cause a related issue where the IDE would try to pull in test/*.cpp for compilation if someone included a library as a dependency.

@ianfixes
Copy link
Author

ianfixes commented Dec 7, 2020

As the arduino_ci GitHub Action takes shape, it becomes more difficult for me to change things later.

If there is any great objection to calling the directory test/ (note singular spelling, not plural), I'd value that feedback as early as you can provide it.

@ianfixes ianfixes linked a pull request Jan 3, 2021 that will close this issue
@ianfixes
Copy link
Author

ianfixes commented Jan 3, 2021

Opened #1125

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: enhancement Proposed improvement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants