What is expected result of tests?

[jgfoster]

System

• OS: macOS Catalina (10.15.6)
• ruby -v: ruby 2.6.3p62 (2019-04-16 revision 67580) [universal.x86_64-darwin19]
• bundle -v: Bundler version 1.17.2
• bundle info arduino_ci:
  • arduino_ci (0.3.0)
    Summary: Tools for building and unit testing Arduino libraries
    Homepage: http://github.com/ianfixes/arduino_ci
    Path: /Users/jfoster/code/arduino_ci
• g++ -v:
  Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
  Apple clang version 11.0.3 (clang-1103.0.32.62)
  Target: x86_64-apple-darwin19.6.0
  Thread model: posix
  InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
• Arduino IDE version: 1.8.13
• java -version:
  java version "11.0.7" 2020-04-14 LTS
  Java(TM) SE Runtime Environment 18.9 (build 11.0.7+8-LTS)
  Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.7+8-LTS, mixed mode)
• URL of failing Travis CI job: N/A
• URL of your Arduino project: https://github.com/Open-Acidification/Open_Acidification_pH-stat_arduino

What is expected result of tests?

I'm looking at arduino_ci for our project and for a class I'm teaching this term. If we do use it I'll likely be making contributions and asking my students to do so as well (if nothing else, to documentation and samples).

I've forked, cloned, and opened a terminal in SampleProjects/DoSomething. When I ran bundle exec arduino_ci_remote.rb, the result was a couple hundred lines of output, including the following:

...Unit testing bad-null.cpp with g++                                          ✗
...Unit testing bad-null.cpp with g++                                          ✗
...Unit testing bad-null.cpp with g++                                          ✗
WARNING: Spurious .bundle folder in 'DoSomething' library
WARNING: Spurious .bundle folder in 'TestSomething' library
fork/exec /Users/jfoster/Arduino.app/Contents/Java/hardware/tools/avr/bin/avr-g++: bad CPU type in executable
                                           ✗
WARNING: Spurious .bundle folder in 'DoSomething' library
WARNING: Spurious .bundle folder in 'TestSomething' library
WARNING: library DoSomething claims to run on (avr) architecture(s) and may be incompatible with your current board which runs on (sam) architecture(s).
fork/exec /Users/jfoster/Arduino.app/Contents/Java/tools-builder/ctags/5.8-arduino11/ctags: bad CPU type in executable
                                           ✗
WARNING: Spurious .bundle folder in 'DoSomething' library
WARNING: Spurious .bundle folder in 'TestSomething' library
fork/exec /Users/jfoster/Arduino.app/Contents/Java/hardware/tools/avr/bin/avr-g++: bad CPU type in executable
                                           ✗
Failures: 6

I'm confused by a few things.

1. I understand that bad-null.cpp is supposed to fail, but my experience with unit testing is that expected failures are reported separately from unexpected failures. Is there a way for an automated script to know that the failures were expected?
2. I assume that the bad CPU type message is because it is trying to run the tests on uno, due, and leonardo. When I remove the three platforms from .arduino-ci.yaml it still reports bad CPU type but doesn't run any tests. How do I run tests locally (in my case, on my Mac), and not try to run them on the three sample platforms?
3. Who is complaining about the Spurious .bundle folder and why? In my case this directory contains a config file with BUNDLE_PATH: "vendor/bundle" so that nothing gets installed with root access.
4. Why is the test looking in the TestSomething directory when I'm in the DoSomething directory?

Arduino or Unit Test Code, Illustrating the Problem

SampleProjects/DoSomething

Arduino Architecture(s) Affected

I'm trying to stay off the Arduino for testing purposes, but my target device is a Mega 2560.

[ianfixes]

Hi James, I'm flattered that you're considering using this project for a class. I'm always looking to improve the documentation; being able to test a project that is itself a test framework has led to the sample projects being tightly coupled to the CI tests of the ruby code.

If you plan to contribute, it might make sense to split this into issues that can be tracked separately:

Expected vs unexpected failures

I understand that bad-null.cpp is supposed to fail, but my experience with unit testing is that expected failures are reported separately from unexpected failures. Is there a way for an automated script to know that the failures were expected?

This is really 2 issues.

How failures are understood by the script to be "expected"

It sounds like the project README could use some clarification about the sample itself

These files are designed to test the Ruby gem itself, such that its basic tasks of library installation and compilation can be verified.

So the failures are only "expected" in this meta sense. In a normal CI scenario for a library, all tests would be expected to pass.

Reporting unexpected failures all at once at the end, separate from expected pass/fails

The difficulty here is that a lot of different operations being reported -- anything whose failure is actionable is presented as a ✔️ or ❌ in the ouput: installation of the test environment, validation of the test environment itself, validation of the configuration, attempts to compile the unit test binaries, actual tests within those binaries (followed often by tests of compilation of sample projects), etc.

Since many of these failures can happen within loops (e.g. when several compiler binaries are specified, entire test suites are repeated), I hadn't come up with a sensible way to put those failures in proper context. Easier to just leave them in the original context and report the total number at the end.

So the real enhancement here is to come up with what the overall textual report format (with all these possibilities) would look like, and then create a data structure to collect & report results in that framework.

bad CPU type

This looks like a problem with Arduino on a 64-bit OSX machine.

The Arduino IDE (well, the toolchain) is 32-bit, Mac OS 10.15 only runs 64-bit applications. Hence, the IDE is currently not compatible with 10.15.

I don't have a 10.15 environment handy to develop a fix for this, and I'm unclear whether a fix will require separate support for pre- and post- 10.15 setups.

Spurious .bundle folder / TestSomething vs DoSomething

If memory serves, this error is from Arduino. In order to have the Arduino IDE compile the library, the folder must exist (in this case via symlink) in the system-wide Arduino library directory. So when the library folder is scanned, any & all warnings for individual libraries are printed. That's what you see here.

[jgfoster]

Thank you for your answers. I believe the issues are properly addressed now.