Spectests code coverage

[chfast]

This creates coverage report (same as for unittests) out of spectests execution. It is available as CircleCI artifact.

The codecov report can be split into two parts, but I'm leaving that for other time.

Closes #406

[codecov]

Codecov Report

Merging #410 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #410   +/-   ##
=======================================
  Coverage   99.17%   99.17%           
=======================================
  Files          49       49           
  Lines       13236    13237    +1     
=======================================
+ Hits        13127    13128    +1     
  Misses        109      109           

[chfast]

First spectests coverage report: https://21739-224892340-gh.circle-artifacts.com/0/coverage/index.html

[gumb0]

The codecov report can be split into two parts, but I'm leaving that for other time.

Do you mean splitting into collecting coverage and uploading to codecov.io? If so, maybe it's better to keep --required till then, otherwise if it fails to upload, it will not be visible that previousle uploaded report was not updated?

[chfast]

Do you mean splitting into collecting coverage and uploading to codecov.io? If so, maybe it's better to keep --required till then, otherwise if it fails to upload, it will not be visible that previousle uploaded report was not updated?

I can keep it but in case of upload failure spectests will not run at all.

By codecov report I mean you can do two uploads - for unittests and spectests with different flags. Then you can filter by flags and have different stats. https://docs.codecov.io/docs/flags

[gumb0]

By codecov report I mean you can do two uploads - for unittests and spectests with different flags. Then you can filter by flags and have different stats. https://docs.codecov.io/docs/flags

This might be a fine plan, but maybe we don't need to collect/upload spectest coverage for each PR? Maybe it could be manually triggered like benchmarks.

[chfast]

This might be a fine plan, but maybe we don't need to collect/upload spectest coverage for each PR? Maybe it could be manually triggered like benchmarks.

It is better to keep it simple and collect every time.