Add generic test coverage report format for SonarQube #1612

mrodm · 2023-12-22T13:40:45Z

Relates #1595

Update Cobertura coverage reports
Add support for Generic test coverage report format for SonarQube
Add new flag to create Generic or Cobertura coverage reports
- Added --coverage-format flag
Add minimal support for coverage input packages
- Assets tests in input packages are not working
- Complete support for input packages still requires more work to do.

PR to test these changes in the integrations repository: elastic/integrations#8464

Example of commands:

# to be backwards compatible, this generates cobertura reports
elastic-package test --test-coverage

# Using the new flag to set the desired format
elastic-package test --test-coverage --coverage-format=cobertura
elastic-package test --test-coverage --coverage-format=generic

IMPORTANT: Introduced a workaround to be able to show coverage for asset, static, system and pipeline (if no ingest pipeline is defined) tests in SonarQube

Use different lines per each test type to show if it is covered or not (if tests exist or not).
- Ensure that it gets represented by SonarQube as a missing or covered line.

Fixes performed in Cobertura format:

Use valid file paths for filename attribute in class element
- Required by SonarQube
Add information about lines in class element too
Remove hits key from CoberturaMethod (not present in the DTD).

Before:

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<coverage line-rate="0" branch-rate="0" version="" timestamp="1705071908664826427" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0" complexity="0">
  <sources></sources>
  <packages>
    <package name="mongodb" line-rate="0" branch-rate="0" complexity="0">
      <classes>
        <class name="system" filename="mongodb/status" line-rate="0" branch-rate="0" complexity="0">
          <methods>
            <method name="OK" signature="" line-rate="0" branch-rate="0" complexity="0" hits="0">
              <lines>
                <line number="1" hits="1"></line>
              </lines>
            </method>
          </methods>
          <lines></lines>
        </class>
      </classes>
    </package>
  </packages>
</coverage>

After using Cobertura:

<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<coverage line-rate="0" branch-rate="0" version="" timestamp="1705071445658692562" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0" complexity="0">
  <sources></sources>
  <packages>
    <package name="mongodb" line-rate="0" branch-rate="0" complexity="0">
      <classes>
        <class name="system" filename="test/packages/parallel/mongodb/data_stream/status/manifest.yml" line-rate="0" branch-rate="0" complexity="0">
          <methods>
            <method name="OK" signature="" line-rate="0" branch-rate="0" complexity="0">
              <lines>
                <line number="3" hits="1"></line>
              </lines>
            </method>
          </methods>
          <lines>
            <line number="3" hits="1"></line>
          </lines>
        </class>
      </classes>
    </package>
  </packages>
</coverage>

After using Generic test format:

<?xml version="1.0" encoding="UTF-8"?>

<coverage version="1">
  <file path="packages/elastic_package_registry/data_stream/metrics/manifest.yml">
    <lineToCover lineNumber="3" covered="true"></lineToCover>
  </file>
  <!--Coverage for system test-->
</coverage>

Pending to load assets for input packages

mrodm · 2024-01-17T11:56:12Z

internal/testrunner/coverageoutput.go

This file has been renamed to be coveragereport.go and part of its logic related to the Cobertura format has been moved to its own file coberturacoverage.go

mrodm · 2024-01-17T15:03:07Z

internal/packages/assets.go

@@ -153,6 +153,8 @@ func loadElasticsearchAssets(pkgRootPath string) ([]Asset, error) {
 		}
 	}

+	// TODO add assets for input packages


Created issue for this: #1623

mrodm · 2024-01-17T15:26:26Z

internal/testrunner/coberturacoverage.go

+type CoberturaMethod struct {
+	Name       string           `xml:"name,attr"`
+	Signature  string           `xml:"signature,attr"`
+	LineRate   float32          `xml:"line-rate,attr"`
+	BranchRate float32          `xml:"branch-rate,attr"`
+	Complexity float32          `xml:"complexity,attr"`
+	Lines      []*CoberturaLine `xml:"lines>line"`
+}


Removed Hits field from this struct since it is not present in the DTD

Do you know if this was used by the jenkins plugin?

I just checked the Jenkins plugin code , looking at the code it does not look like it was used. And their tests do not include that field neither

https://github.com/jenkinsci/cobertura-plugin/blob/ff0973461f60c28dbc0bf5ab4853a5aec4798210/src/test/resources/hudson/plugins/cobertura/coverage-with-data.xml

jsoriano

👍

cmd/testrunner.go

jsoriano · 2024-01-18T07:55:03Z

internal/testrunner/coberturacoverage.go

+type CoberturaMethod struct {
+	Name       string           `xml:"name,attr"`
+	Signature  string           `xml:"signature,attr"`
+	LineRate   float32          `xml:"line-rate,attr"`
+	BranchRate float32          `xml:"branch-rate,attr"`
+	Complexity float32          `xml:"complexity,attr"`
+	Lines      []*CoberturaLine `xml:"lines>line"`
+}


Do you know if this was used by the jenkins plugin?

internal/testrunner/coveragereport.go

cmd/testrunner.go

jsoriano

👍

elasticmachine · 2024-01-18T12:44:20Z

💚 Build Succeeded

Buildkite Build
Commit: 499953c

History

💚 Build #2132 succeeded 5579da4
💚 Build #2131 succeeded 549ffda
💚 Build #2126 succeeded 9b65654
💚 Build #2125 succeeded 23701cd
💔 Build #2124 failed c961c39
💚 Build #2123 succeeded 41531ac

cc @mrodm

This was referenced Dec 22, 2023

add coverage reports elastic/integrations#8464

Closed

Add SonarQube generic-test-data format support for code coverage report. #1595

Closed

mrodm added 9 commits January 12, 2024 13:11

Update paths coverage

c7bf4c9

Add manifest.yml into the path

bd5a500

Add base folder

25cd56f

Get filepath with respect to working copy root

d7c89be

Fix linting

64dfd9e

Remove Hits from Methods attribute not present in DTD

bba2f9b

Add base folder in package name

73c0dfe

Add different lines in manifest.yml for each type of test

af49010

Add lines into CoberturaClass

9cb94d8

mrodm force-pushed the update_paths_coverage_reports branch from 018e2d4 to 9cb94d8 Compare January 12, 2024 12:11

mrodm added 18 commits January 12, 2024 13:13

Check coverage xml files

26ffdac

Remove hits from CoberturaMethod in tests

d584340

Replace / chars in paths for package names

d6e001e

Avoid setting sources for test coverage pipeline

e9e9826

Add generic output - WIP

83ac6bf

Some refactors

82dc88f

Show tests for input packages

811e2cd

Pending to load assets for input packages

Refactor and tests

46ef78b

Refactor

00612aa

Ensure data streams are sorted - more tests added

738fc69

Add comments

522ffd6

Manage coverage paths for windows

9d153d3

Remove base folder from CoberturaPackage

9193fd9

Use filepath for tests

0a6eaee

Update method to get base folder using filepath

f938f12

Use filepath for files in cverage reports

41531ac

Add package type as part of the coverage report file name

c961c39

Add test type as part of the coverage report file name

23701cd

mrodm commented Jan 17, 2024

View reviewed changes

mrodm added 4 commits January 17, 2024 12:59

Remove leftover from testing

a51cb2f

Remove debug leftovers

f667f8c

Ensure right paths for sources and filenames (pipeline coverage)

d98f7b4

Rephrase comments

9b65654

mrodm marked this pull request as ready for review January 17, 2024 15:01

mrodm requested a review from a team January 17, 2024 15:02

mrodm self-assigned this Jan 17, 2024

mrodm changed the title ~~Update paths coverage~~ Add generic test coverage report format for SonarQube Jan 17, 2024

mrodm commented Jan 17, 2024

View reviewed changes

jsoriano approved these changes Jan 18, 2024

View reviewed changes

jsoriano reviewed Jan 18, 2024

View reviewed changes

internal/testrunner/coveragereport.go Outdated Show resolved Hide resolved

cmd/testrunner.go Outdated Show resolved Hide resolved

cmd/testrunner.go Outdated Show resolved Hide resolved

mrodm added 2 commits January 18, 2024 10:46

Rename createCoverageReport parameter

aa9e30c

Check coverage format value set in the parameter

549ffda

jsoriano approved these changes Jan 18, 2024

View reviewed changes

mrodm added 2 commits January 18, 2024 11:40

Update test scripts to use coverage format generic

5579da4

Add missing version field

499953c

jsoriano approved these changes Jan 18, 2024

View reviewed changes

mrodm merged commit 282cfe3 into elastic:main Jan 18, 2024
3 checks passed

mrodm deleted the update_paths_coverage_reports branch January 18, 2024 12:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add generic test coverage report format for SonarQube #1612

Add generic test coverage report format for SonarQube #1612

mrodm commented Dec 22, 2023 •

edited

Loading

mrodm Jan 17, 2024

mrodm Jan 17, 2024

mrodm Jan 17, 2024 •

edited

Loading

jsoriano Jan 18, 2024

mrodm Jan 18, 2024

jsoriano left a comment

jsoriano Jan 18, 2024

jsoriano left a comment

elasticmachine commented Jan 18, 2024

Add generic test coverage report format for SonarQube #1612

Add generic test coverage report format for SonarQube #1612

Conversation

mrodm commented Dec 22, 2023 • edited Loading

mrodm Jan 17, 2024

Choose a reason for hiding this comment

mrodm Jan 17, 2024

Choose a reason for hiding this comment

mrodm Jan 17, 2024 • edited Loading

Choose a reason for hiding this comment

jsoriano Jan 18, 2024

Choose a reason for hiding this comment

mrodm Jan 18, 2024

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano Jan 18, 2024

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

elasticmachine commented Jan 18, 2024

💚 Build Succeeded

History

mrodm commented Dec 22, 2023 •

edited

Loading

mrodm Jan 17, 2024 •

edited

Loading