Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Code coverage for e2e tests #1475

Merged
merged 6 commits into from
Oct 7, 2019
Merged

Conversation

rewenset
Copy link
Contributor

@rewenset rewenset commented Sep 23, 2019

Created a dummy test that executes main() of vpp-agent. Each e2e test starts that executable with enabled coverage analysis. All coverage data will be stored in /tmp/e2e-coverage directory and after merged with gocovmerge to one /tmp/e2e-all.out file.

The idea of how coverage could be collected was borrowed from "Go coverage with external tests".

@rewenset rewenset added the 🚧 WIP do not merge! work in progress! label Sep 23, 2019
@rewenset
Copy link
Contributor Author

rewenset commented Sep 24, 2019

This is a coverage of all e2e tests:

Total Coverage: 32.23% (13891/43101)

EDIT:
And if I go and check coverage of each vpp-agent run:

Total Coverage: 24.65% (10625/43101)
Total Coverage: 25.43% (10960/43101)
Total Coverage: 29.69% (12797/43101)
Total Coverage: 29.60% (12757/43101)
Total Coverage: 28.64% (12345/43101)
Total Coverage: 30.96% (13342/43101)
Total Coverage: 31.01% (13364/43101)
Total Coverage: 29.33% (12643/43101)

Also, for example, test TestAgentInSync in tests/e2e/000_initial_test.go produces:

coverage: 24.7% of statements in github.com/ligato/vpp-agent/...

and as for test TestTapInterfaceConn:

coverage: 29.7% of statements in github.com/ligato/vpp-agent/...

Also I'm able to generate html file, but not sure how to share it :)
image

@ondrej-fabry
Copy link
Member

ondrej-fabry commented Sep 24, 2019

This is a coverage of all e2e tests:

Total Coverage: 32.23% (13891/43101)

What are those numbers on the right? Is it really counting all packages?
Can you try to find list of all packages and their respective coverage?

@rewenset
Copy link
Contributor Author

rewenset commented Sep 25, 2019

I'm using gocov for coverage analysis.

  1. Those numbers are (reached statements / total statements).
  2. Yes, seems it is
  3. List of all packages will be huge, but I can dump here part of it:
    (also I've intentionally put there coverage of cmd/vpp-agent/, so you may notice it is also included)
Packages coverage after running all E2E tests

I've removed all - with bash sed tool, because gocov generates summary about package with annoying dashes hyphens

[...]
github.com/ligato/vppagent/cmd/vppagent     34.48% (10/29)                           
github.com/ligato/vppagent/cmd/vppagent/app      100.00% (28/28)              
github.com/ligato/vppagent/pkg/debug     0.00% (0/23)                                
github.com/ligato/vppagent/pkg/idxvpp     54.29% (19/35)                              
github.com/ligato/vppagent/pkg/metrics      60.00% (9/15)                             
github.com/ligato/vppagent/pkg/models     48.90% (89/182)                      
github.com/ligato/vppagent/pkg/util     0.00% (0/33)                          
github.com/ligato/vppagent/plugins/configurator     18.55% (59/318)               
github.com/ligato/vppagent/plugins/govppmux       67.47% (195/289)           
github.com/ligato/vppagent/plugins/govppmux/vppcalls      72.73% (8/11)       
github.com/ligato/vppagent/plugins/govppmux/vppcalls/vpp1904      11.43% (4/35)
github.com/ligato/vppagent/plugins/govppmux/vppcalls/vpp1908      80.00% (28/35)      
github.com/ligato/vppagent/plugins/govppmux/vppcalls/vpp2001      11.76% (4/34)
github.com/ligato/vppagent/plugins/kvscheduler       62.22% (1242/1996)        
github.com/ligato/vppagent/plugins/kvscheduler/api      52.94% (144/272)          
github.com/ligato/vppagent/plugins/kvscheduler/internal/graph      64.23% (598/931)
github.com/ligato/vppagent/plugins/kvscheduler/internal/registry      85.00% (34/40)
github.com/ligato/vppagent/plugins/kvscheduler/internal/utils      51.57% (164/318)     
github.com/ligato/vppagent/plugins/linux/ifplugin     72.58% (45/62)
github.com/ligato/vppagent/plugins/linux/ifplugin/descriptor       58.50% (444/759)
github.com/ligato/vppagent/plugins/linux/ifplugin/descriptor/adapter      73.20% (71/97)
github.com/ligato/vppagent/plugins/linux/ifplugin/ifaceidx     51.28% (20/39)
github.com/ligato/vppagent/plugins/linux/ifplugin/linuxcalls       69.44% (75/108)
github.com/ligato/vppagent/plugins/linux/iptablesplugin      67.50% (27/40)
github.com/ligato/vppagent/plugins/linux/iptablesplugin/descriptor     13.12% (21/160)
github.com/ligato/vppagent/plugins/linux/iptablesplugin/descriptor/adapter     25.77% (25/97)
github.com/ligato/vppagent/plugins/linux/iptablesplugin/linuxcalls       16.98% (9/53)
github.com/ligato/vppagent/plugins/linux/l3plugin    71.43% (30/42)
github.com/ligato/vppagent/plugins/linux/l3plugin/descriptor     73.05% (225/308)
github.com/ligato/vppagent/plugins/linux/l3plugin/descriptor/adapter     60.82% (118/194)
github.com/ligato/vppagent/plugins/linux/l3plugin/linuxcalls       86.67% (13/15)
github.com/ligato/vppagent/plugins/linux/nsplugin     63.56% (75/118)
github.com/ligato/vppagent/plugins/linux/nsplugin/descriptor      84.96% (113/133)
github.com/ligato/vppagent/plugins/linux/nsplugin/linuxcalls       19.75% (16/81)
github.com/ligato/vppagent/plugins/netalloc      79.76% (67/84)
github.com/ligato/vppagent/plugins/netalloc/descriptor     92.31% (24/26)
github.com/ligato/vppagent/plugins/netalloc/descriptor/adapter      54.64% (53/97)
github.com/ligato/vppagent/plugins/netalloc/utils      74.60% (47/63)
github.com/ligato/vppagent/plugins/orchestrator      40.41% (118/292)
github.com/ligato/vppagent/plugins/restapi      37.85% (137/362)
github.com/ligato/vppagent/plugins/telemetry      77.73% (171/220)
github.com/ligato/vppagent/plugins/telemetry/vppcalls      54.84% (17/31)
github.com/ligato/vppagent/plugins/telemetry/vppcalls/vpp1904       2.65% (4/151)
github.com/ligato/vppagent/plugins/telemetry/vppcalls/vpp1908       48.41% (76/157)
github.com/ligato/vppagent/plugins/telemetry/vppcalls/vpp2001       2.55% (4/157)
github.com/ligato/vppagent/plugins/vpp/abfplugin     77.14% (27/35)
github.com/ligato/vppagent/plugins/vpp/abfplugin/abfidx     9.52% (2/21)
github.com/ligato/vppagent/plugins/vpp/abfplugin/descriptor      12.90% (12/93)
github.com/ligato/vppagent/plugins/vpp/abfplugin/descriptor/adapter    26.80% (26/97)
github.com/ligato/vppagent/plugins/vpp/abfplugin/vppcalls     77.78% (7/9)
[...]

Or coverage of registry package from internal of kvscheduler plugin

github.com/ligato/vpp-agent/plugins/kvscheduler/internal/registry/registry_impl.go       registry.GetAllDescriptors      100.00% (13/13)
github.com/ligato/vpp-agent/plugins/kvscheduler/internal/registry/registry_impl.go       registry.RegisterDescriptor     100.00% (2/2)
github.com/ligato/vpp-agent/plugins/kvscheduler/internal/registry/registry_impl.go       NewRegistry                     100.00% (1/1)
github.com/ligato/vpp-agent/plugins/kvscheduler/internal/registry/registry_impl.go       registry.GetDescriptorForKey    75.00% (15/20)
github.com/ligato/vpp-agent/plugins/kvscheduler/internal/registry/registry_impl.go       registry.GetDescriptor          75.00% (3/4)
github.com/ligato/vpp-agent/plugins/kvscheduler/internal/registry                        ----------------------------    85.00% (34/40)

[...]

@rewenset rewenset changed the title WIP: Code coverage for e2e tests test: Code coverage for e2e tests Sep 27, 2019
@rewenset rewenset changed the title test: Code coverage for e2e tests test: Code coverage for e2e tests WIP Sep 27, 2019
@codecov
Copy link

codecov bot commented Sep 27, 2019

Codecov Report

❗ No coverage uploaded for pull request base (dev@7ec11e5). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@          Coverage Diff           @@
##             dev    #1475   +/-   ##
======================================
  Coverage       ?   52.12%           
======================================
  Files          ?      544           
  Lines          ?    65827           
  Branches       ?        0           
======================================
  Hits           ?    34314           
  Misses         ?    29274           
  Partials       ?     2239
Flag Coverage Δ
#e2e1904 35.93% <ø> (?)
#e2e1908 36.16% <ø> (?)
#e2e2001 36.27% <ø> (?)
#unittests 52.85% <ø> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7ec11e5...527ca03. Read the comment docs.

@rewenset rewenset changed the title test: Code coverage for e2e tests WIP test: Code coverage for e2e tests Sep 27, 2019
@rewenset rewenset removed the 🚧 WIP do not merge! work in progress! label Sep 27, 2019
@rewenset
Copy link
Contributor Author

upload unittests coverage to coveralls was intentionally left there. I think that it would be better to remove it after some time with codecov

@ondrej-fabry ondrej-fabry merged commit 18b6a08 into ligato:dev Oct 7, 2019
VladoLavor pushed a commit to VladoLavor/vpp-agent that referenced this pull request Oct 18, 2019
* Initial collecting coverage for e2e tests

* Allow to run e2e tests with and without coverage

* Add build tag to dummy test, so it is skipped in general case

* Upload coverage to codecov

test-e2e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants