github-actions: add codecov upload

[miri64]

This adds an upload of the coverage report to codecov. This way, we can track if a PR increases or decreases test coverage.

Requires #2 to be mergable into protected branch 😅

[miri64]

Rebased to current master and squashed comments

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request head (gh-actions/enh/codecov-upload@eb339aa). Click here to learn what that means.
The diff coverage is n/a.

[miri64]

@aabadie The coverage is shown empty now even though there is a coverage report 😕:

• https://github.com/RIOT-OS/riotctrl/pull/1/checks?check_run_id=844708477#step:5:44
• https://github.com/RIOT-OS/riotctrl/pull/1/checks?check_run_id=844708477#step:6:27

[aabadie]

Maybe try with a new version of the codecov action. On riot-generator, the version is 1.0.7.

[miri64]

I have the feeling this might be because it is not in the master branch :-/.

[miri64]

From the PR's report

#1 updated by miri64 Missing base report.

[miri64]

I deactivated branch protection and directly pushed to master to test my hypothesis. Will clean-up after.

[miri64]

(pushing to a test branch did not work as that branch then is ignored by Github Actions)

[miri64]

It's an issue with the codecov-action workflow -.- codecov/codecov-action#85. I'll apply the workaround there in an extra commit.

[miri64]

Still doesn't work :(

[miri64]

https://codecov.io/gh/RIOT-OS/riotctrl/commit/eb339aa08d66f50f8bd4db4d29e6fedb04cefdec

[miri64]

Let's give this another chance

[codecov-io]

Codecov Report

Merging #1 (413badb) into master (c141a9b) will increase coverage by 19.33%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##           master        #1       +/-   ##
============================================
+ Coverage   80.66%   100.00%   +19.33%     
============================================
  Files           9         2        -7     
  Lines         269        60      -209     
  Branches        0         4        +4     
============================================
- Hits          217        60      -157     
+ Misses         52         0       -52     

Flag	Coverage Δ
3.5	?
3.6	98.30% <96.15%> (+17.63%)	⬆️
3.7	98.30% <96.15%> (+17.63%)	⬆️
3.8	98.33% <96.15%> (?)
3.9	98.33% <96.15%> (?)
test	98.33% <96.15%> (?)
test-rapidjson	61.66% <76.92%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
riotctrl/shell/__init__.py	100.00% <100.00%> (ø)
riotctrl/shell/json.py	100.00% <100.00%> (ø)
riotctrl/tests/utils/application/echo.py
riotctrl/tests/utils/application/create_file.py
riotctrl/tests/utils/application/ctrl.py
riotctrl/tests/riotctrl_test.py
riotctrl/tests/utils/application/sigkill_script.py
riotctrl/tests/utils/application/shell.py
riotctrl/tests/utils/application/hello.py
riotctrl/tests/ctrl_test.py
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b30a25...413badb. Read the comment docs.

[miri64]

That seemed to have triggered something. At least now it does not insist on measuring the coverage of the tests and nothing of the module. ^^

[miri64]

Rebased and squashed to current master.

[miri64]

Rebased and squashed to current master.

Again.

[fjmolinas]

Rebased and squashed to current master.

Again.

How to I verify this is working?

[miri64]

Appearently it is not working at all, atm, see current build status. Also, I am not sure how @aabadie's recent tox rework factors into this... :-/

[codecov-commenter]

Codecov Report

Merging #1 (89d86af) into master (c141a9b) will increase coverage by 15.28%.
The diff coverage is 94.87%.

@@             Coverage Diff             @@
##           master       #1       +/-   ##
===========================================
+ Coverage   80.66%   95.95%   +15.28%     
===========================================
  Files           9        4        -5     
  Lines         269      173       -96     
  Branches        0       11       +11     
===========================================
- Hits          217      166       -51     
+ Misses         52        6       -46     
- Partials        0        1        +1     

Flag	Coverage Δ
3.5	?
3.6	?
3.7	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
riotctrl/ctrl.py	93.45% <90.69%> (ø)
riotctrl/__init__.py	100.00% <100.00%> (ø)
riotctrl/shell/__init__.py	100.00% <100.00%> (ø)
riotctrl/shell/json.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee12bb5...89d86af. Read the comment docs.

[miri64]

That's exactly as I feared: Now it is only showing the coverage results of rapidjson, the testenv that ran last.

[aabadie]

Normally codecov is supposed to merge results from different testenv (that's one of it's strength).

[miri64]

Normally codecov is supposed to merge results from different testenv (that's one of it's strength).

But only if you use --cov-append with pytest-cov. Did that in ef0f6d4

[aabadie]

Why do need to add these lines ? In theory, these kind of things could be tested.

[miri64]

Why do need to add these lines ? In theory, these kind of things could be tested.

I was called Mrs "100% percent coverage" by @MrKevinWeiss before, but even I don't test code that states on its own that it is not implemented ;-).

[aabadie]

I think this is a bit out of scope of this PR anyway. And I think, this only concerns

riotctrl/riotctrl/ctrl.py

Line 237 in ee12bb5

raise NotImplementedError

The "raise NotImplementedError" is useless since it's an abstract method, you could just replace it with .... When instanciating a concrete class that doesn't implement the get_ctrl method, python will raise a TypeError exception telling what method is missing.

[aabadie]

To summarize, I think this whole configuration block should be removed.

[miri64]

I think this is a bit out of scope of this PR anyway. And I think, this only concerns

riotctrl/riotctrl/ctrl.py

Line 237 in ee12bb5

raise NotImplementedError

But then get a missing coverage there....

[aabadie]

then get a missing coverage there...

Then just replace this line with ... # pragma: no cover

If you want to ensure a derived class cannot be instanciated without a proper implementation of the get_ctrl method, you can use this test:

def test_not_implemented_factory():
    class MyFactory(riotctrl.ctrl.RIOTCtrlFactoryBase):
        pass

    with pytest.raises(TypeError) as exc_info:
        _ = MyFactory()

    assert exc_info.value.args[0] == (
        "Can't instantiate abstract class MyFactory "
        "with abstract method get_ctrl"
    )

but you definitely don't need to add an exclude_lines config in setup.cfg.

[miri64]

Fine... let's finally get this in and worry about actual coverage later...