Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Coverage for pipeline tests are not generated for all data streams in some scenarios #1624

Closed
mrodm opened this issue Jan 17, 2024 · 0 comments · Fixed by #1915
Closed

Coverage for pipeline tests are not generated for all data streams in some scenarios #1624

mrodm opened this issue Jan 17, 2024 · 0 comments · Fixed by #1915

Comments

@mrodm
Copy link
Contributor

mrodm commented Jan 17, 2024 •
edited
Loading

Relates #1595

Currently, there are three scenarios for reporting coverage in pipeline tests considering
just one data stream:

  1. No ingest pipeline defined (and no tests)
    • No report is generated, empty coverage file. 
      <?xml version="1.0" encoding="UTF-8"?>

<coverage version="1">
  <!--Coverage for pipeline test--></coverage>
  2. There is an ingest pipeline but no tests
    • It creates a coverage report with a Missing method* 
      <?xml version="1.0" encoding="UTF-8"?>

<coverage version="1">
  <file path="packages/elastic_package_registry/data_stream/metrics/manifest.yml">
    <lineToCover lineNumber="2" covered="false"></lineToCover>
  </file>
  <!--Coverage for pipeline test-->
</coverage>
  3. There is an ingest pipeline and tests for that pipeline
    • It creates a coverage report with all the processors used as methods 
      <?xml version="1.0" encoding="UTF-8"?>

<coverage version="0">
  <file path="packages/elastic_package_registry/data_stream/metrics/elasticsearch/ingest_pipeline/default.yml">
    <lineToCover lineNumber="4" covered="true"></lineToCover>
    <lineToCover lineNumber="5" covered="true"></lineToCover>
    <lineToCover lineNumber="6" covered="true"></lineToCover>
    <lineToCover lineNumber="10" covered="true"></lineToCover>
....
    <lineToCover lineNumber="85" covered="true"></lineToCover>
    <lineToCover lineNumber="86" covered="true"></lineToCover>
    <lineToCover lineNumber="87" covered="true"></lineToCover>
    <lineToCover lineNumber="88" covered="true"></lineToCover>
  </file>
  <!--Cobertura for pipeline test-->
</coverage>

This is working correctly while there is just one data stream with ingest pipelines.
If there are multiple data streams and not all of them with ingest pipelines, there is an scenario not covered. As an example:

  • Package containing 2 data streams
  • Both data streams have an ingest pipeline defined
  • Just one data stream has pipeline tests defined for the ingest pipelines.

In this case, the coverage report will just show the coverage report for the data stream
with tests, but it will not add any reference for the second data stream that has no tests. It should also add a coverage reference adding a Missing (not covered) method/line.

For instance , this happens with the hashicorp_vault package in the integrations repository:
https://github.com/elastic/integrations/tree/f3141bc947df480b628788c01ba50fdc411a22b0/packages/hashicorp_vault/data_stream

  • This package has 3 data streams
  • There are ingest pipelines defined in those 3 data streams
  • Just one data stream has no tests for pipeline

It just contains the coverage information for audit and log but there is no reference to the third data stream (ingest pipeline) adding a not covered line as mentioned above.

Coverage XML generated for hashicorp_vault package (pipeline tests) 
<?xml version="1.0" encoding="UTF-8"?>

<coverage version="0">
  <file path="packages/hashicorp_vault/data_stream/audit/elasticsearch/ingest_pipeline/default.yml">
    <lineToCover lineNumber="4" covered="true"></lineToCover>
    <lineToCover lineNumber="5" covered="true"></lineToCover>
    <lineToCover lineNumber="6" covered="true"></lineToCover>
    <lineToCover lineNumber="7" covered="true"></lineToCover>
    <lineToCover lineNumber="8" covered="true"></lineToCover>
    <lineToCover lineNumber="9" covered="true"></lineToCover>
    <lineToCover lineNumber="10" covered="true"></lineToCover>
    <lineToCover lineNumber="11" covered="true"></lineToCover>
    <lineToCover lineNumber="12" covered="true"></lineToCover>
    <lineToCover lineNumber="13" covered="true"></lineToCover>
    <lineToCover lineNumber="14" covered="true"></lineToCover>
    <lineToCover lineNumber="15" covered="true"></lineToCover>
    <lineToCover lineNumber="16" covered="true"></lineToCover>
    <lineToCover lineNumber="21" covered="true"></lineToCover>
    <lineToCover lineNumber="22" covered="true"></lineToCover>
    <lineToCover lineNumber="23" covered="true"></lineToCover>
    <lineToCover lineNumber="24" covered="true"></lineToCover>
    <lineToCover lineNumber="25" covered="true"></lineToCover>
    <lineToCover lineNumber="26" covered="true"></lineToCover>
    <lineToCover lineNumber="27" covered="true"></lineToCover>
    <lineToCover lineNumber="29" covered="true"></lineToCover>
    <lineToCover lineNumber="33" covered="true"></lineToCover>
    <lineToCover lineNumber="34" covered="true"></lineToCover>
    <lineToCover lineNumber="35" covered="true"></lineToCover>
    <lineToCover lineNumber="36" covered="true"></lineToCover>
    <lineToCover lineNumber="37" covered="true"></lineToCover>
    <lineToCover lineNumber="38" covered="true"></lineToCover>
    <lineToCover lineNumber="40" covered="false"></lineToCover>
    <lineToCover lineNumber="41" covered="false"></lineToCover>
    <lineToCover lineNumber="42" covered="false"></lineToCover>
    <lineToCover lineNumber="43" covered="false"></lineToCover>
    <lineToCover lineNumber="44" covered="false"></lineToCover>
    <lineToCover lineNumber="45" covered="true"></lineToCover>
    <lineToCover lineNumber="46" covered="true"></lineToCover>
    <lineToCover lineNumber="47" covered="true"></lineToCover>
    <lineToCover lineNumber="48" covered="true"></lineToCover>
    <lineToCover lineNumber="49" covered="true"></lineToCover>
    <lineToCover lineNumber="50" covered="true"></lineToCover>
    <lineToCover lineNumber="51" covered="true"></lineToCover>
    <lineToCover lineNumber="52" covered="true"></lineToCover>
    <lineToCover lineNumber="53" covered="true"></lineToCover>
    <lineToCover lineNumber="54" covered="true"></lineToCover>
    <lineToCover lineNumber="55" covered="true"></lineToCover>
    <lineToCover lineNumber="56" covered="true"></lineToCover>
    <lineToCover lineNumber="57" covered="true"></lineToCover>
    <lineToCover lineNumber="58" covered="true"></lineToCover>
    <lineToCover lineNumber="59" covered="true"></lineToCover>
    <lineToCover lineNumber="60" covered="true"></lineToCover>
    <lineToCover lineNumber="61" covered="true"></lineToCover>
    <lineToCover lineNumber="62" covered="true"></lineToCover>
    <lineToCover lineNumber="63" covered="true"></lineToCover>
    <lineToCover lineNumber="64" covered="true"></lineToCover>
    <lineToCover lineNumber="65" covered="true"></lineToCover>
    <lineToCover lineNumber="66" covered="true"></lineToCover>
    <lineToCover lineNumber="67" covered="true"></lineToCover>
    <lineToCover lineNumber="68" covered="true"></lineToCover>
    <lineToCover lineNumber="73" covered="true"></lineToCover>
    <lineToCover lineNumber="74" covered="true"></lineToCover>
    <lineToCover lineNumber="75" covered="true"></lineToCover>
    <lineToCover lineNumber="76" covered="true"></lineToCover>
    <lineToCover lineNumber="77" covered="true"></lineToCover>
    <lineToCover lineNumber="78" covered="true"></lineToCover>
    <lineToCover lineNumber="79" covered="true"></lineToCover>
    <lineToCover lineNumber="80" covered="true"></lineToCover>
    <lineToCover lineNumber="82" covered="true"></lineToCover>
    <lineToCover lineNumber="83" covered="true"></lineToCover>
    <lineToCover lineNumber="84" covered="true"></lineToCover>
    <lineToCover lineNumber="85" covered="true"></lineToCover>
    <lineToCover lineNumber="86" covered="true"></lineToCover>
    <lineToCover lineNumber="87" covered="true"></lineToCover>
    <lineToCover lineNumber="88" covered="true"></lineToCover>
    <lineToCover lineNumber="89" covered="true"></lineToCover>
    <lineToCover lineNumber="90" covered="true"></lineToCover>
    <lineToCover lineNumber="91" covered="true"></lineToCover>
    <lineToCover lineNumber="92" covered="true"></lineToCover>
    <lineToCover lineNumber="93" covered="true"></lineToCover>
    <lineToCover lineNumber="94" covered="true"></lineToCover>
    <lineToCover lineNumber="95" covered="true"></lineToCover>
    <lineToCover lineNumber="96" covered="true"></lineToCover>
    <lineToCover lineNumber="97" covered="true"></lineToCover>
    <lineToCover lineNumber="98" covered="true"></lineToCover>
    <lineToCover lineNumber="99" covered="true"></lineToCover>
    <lineToCover lineNumber="102" covered="true"></lineToCover>
    <lineToCover lineNumber="103" covered="true"></lineToCover>
    <lineToCover lineNumber="104" covered="true"></lineToCover>
    <lineToCover lineNumber="105" covered="true"></lineToCover>
    <lineToCover lineNumber="106" covered="true"></lineToCover>
    <lineToCover lineNumber="107" covered="true"></lineToCover>
    <lineToCover lineNumber="108" covered="true"></lineToCover>
    <lineToCover lineNumber="109" covered="true"></lineToCover>
    <lineToCover lineNumber="112" covered="true"></lineToCover>
    <lineToCover lineNumber="113" covered="true"></lineToCover>
    <lineToCover lineNumber="114" covered="true"></lineToCover>
    <lineToCover lineNumber="115" covered="true"></lineToCover>
    <lineToCover lineNumber="116" covered="true"></lineToCover>
    <lineToCover lineNumber="117" covered="true"></lineToCover>
    <lineToCover lineNumber="118" covered="true"></lineToCover>
    <lineToCover lineNumber="119" covered="true"></lineToCover>
    <lineToCover lineNumber="120" covered="true"></lineToCover>
    <lineToCover lineNumber="121" covered="true"></lineToCover>
    <lineToCover lineNumber="122" covered="true"></lineToCover>
    <lineToCover lineNumber="123" covered="true"></lineToCover>
    <lineToCover lineNumber="124" covered="true"></lineToCover>
    <lineToCover lineNumber="125" covered="true"></lineToCover>
    <lineToCover lineNumber="126" covered="true"></lineToCover>
    <lineToCover lineNumber="127" covered="true"></lineToCover>
    <lineToCover lineNumber="128" covered="true"></lineToCover>
    <lineToCover lineNumber="129" covered="true"></lineToCover>
    <lineToCover lineNumber="130" covered="true"></lineToCover>
    <lineToCover lineNumber="131" covered="true"></lineToCover>
    <lineToCover lineNumber="132" covered="true"></lineToCover>
    <lineToCover lineNumber="133" covered="true"></lineToCover>
    <lineToCover lineNumber="134" covered="true"></lineToCover>
    <lineToCover lineNumber="135" covered="true"></lineToCover>
    <lineToCover lineNumber="136" covered="true"></lineToCover>
    <lineToCover lineNumber="137" covered="true"></lineToCover>
    <lineToCover lineNumber="138" covered="true"></lineToCover>
    <lineToCover lineNumber="139" covered="true"></lineToCover>
    <lineToCover lineNumber="140" covered="true"></lineToCover>
    <lineToCover lineNumber="141" covered="true"></lineToCover>
    <lineToCover lineNumber="142" covered="true"></lineToCover>
    <lineToCover lineNumber="143" covered="true"></lineToCover>
    <lineToCover lineNumber="144" covered="true"></lineToCover>
    <lineToCover lineNumber="145" covered="true"></lineToCover>
    <lineToCover lineNumber="146" covered="true"></lineToCover>
    <lineToCover lineNumber="147" covered="true"></lineToCover>
    <lineToCover lineNumber="150" covered="true"></lineToCover>
    <lineToCover lineNumber="151" covered="true"></lineToCover>
    <lineToCover lineNumber="152" covered="true"></lineToCover>
    <lineToCover lineNumber="153" covered="true"></lineToCover>
    <lineToCover lineNumber="155" covered="true"></lineToCover>
    <lineToCover lineNumber="156" covered="true"></lineToCover>
    <lineToCover lineNumber="157" covered="true"></lineToCover>
    <lineToCover lineNumber="158" covered="true"></lineToCover>
    <lineToCover lineNumber="159" covered="true"></lineToCover>
    <lineToCover lineNumber="160" covered="true"></lineToCover>
    <lineToCover lineNumber="161" covered="true"></lineToCover>
    <lineToCover lineNumber="162" covered="true"></lineToCover>
    <lineToCover lineNumber="163" covered="true"></lineToCover>
    <lineToCover lineNumber="164" covered="true"></lineToCover>
    <lineToCover lineNumber="165" covered="true"></lineToCover>
    <lineToCover lineNumber="166" covered="true"></lineToCover>
    <lineToCover lineNumber="167" covered="true"></lineToCover>
    <lineToCover lineNumber="168" covered="true"></lineToCover>
    <lineToCover lineNumber="169" covered="true"></lineToCover>
    <lineToCover lineNumber="170" covered="true"></lineToCover>
    <lineToCover lineNumber="172" covered="true"></lineToCover>
    <lineToCover lineNumber="173" covered="true"></lineToCover>
    <lineToCover lineNumber="174" covered="true"></lineToCover>
    <lineToCover lineNumber="175" covered="true"></lineToCover>
    <lineToCover lineNumber="176" covered="true"></lineToCover>
    <lineToCover lineNumber="181" covered="false"></lineToCover>
    <lineToCover lineNumber="182" covered="false"></lineToCover>
    <lineToCover lineNumber="183" covered="false"></lineToCover>
    <lineToCover lineNumber="184" covered="false"></lineToCover>
    <lineToCover lineNumber="185" covered="false"></lineToCover>
  </file>
  <file path="packages/hashicorp_vault/data_stream/log/elasticsearch/ingest_pipeline/default.yml">
    <lineToCover lineNumber="4" covered="true"></lineToCover>
    <lineToCover lineNumber="5" covered="true"></lineToCover>
    <lineToCover lineNumber="6" covered="true"></lineToCover>
    <lineToCover lineNumber="7" covered="true"></lineToCover>
    <lineToCover lineNumber="8" covered="true"></lineToCover>
    <lineToCover lineNumber="9" covered="true"></lineToCover>
    <lineToCover lineNumber="10" covered="true"></lineToCover>
    <lineToCover lineNumber="11" covered="true"></lineToCover>
    <lineToCover lineNumber="12" covered="true"></lineToCover>
    <lineToCover lineNumber="13" covered="true"></lineToCover>
    <lineToCover lineNumber="14" covered="true"></lineToCover>
    <lineToCover lineNumber="15" covered="true"></lineToCover>
    <lineToCover lineNumber="19" covered="true"></lineToCover>
    <lineToCover lineNumber="20" covered="true"></lineToCover>
    <lineToCover lineNumber="21" covered="true"></lineToCover>
    <lineToCover lineNumber="25" covered="false"></lineToCover>
    <lineToCover lineNumber="26" covered="false"></lineToCover>
    <lineToCover lineNumber="27" covered="false"></lineToCover>
    <lineToCover lineNumber="28" covered="false"></lineToCover>
    <lineToCover lineNumber="29" covered="false"></lineToCover>
    <lineToCover lineNumber="33" covered="false"></lineToCover>
    <lineToCover lineNumber="34" covered="false"></lineToCover>
    <lineToCover lineNumber="35" covered="false"></lineToCover>
    <lineToCover lineNumber="36" covered="false"></lineToCover>
    <lineToCover lineNumber="37" covered="false"></lineToCover>
  </file>
  <file path="packages/hashicorp_vault/data_stream/log/elasticsearch/ingest_pipeline/json.yml">
    <lineToCover lineNumber="4" covered="true"></lineToCover>
    <lineToCover lineNumber="5" covered="true"></lineToCover>
    <lineToCover lineNumber="6" covered="true"></lineToCover>
    <lineToCover lineNumber="11" covered="true"></lineToCover>
    <lineToCover lineNumber="12" covered="true"></lineToCover>
    <lineToCover lineNumber="13" covered="true"></lineToCover>
    <lineToCover lineNumber="14" covered="true"></lineToCover>
    <lineToCover lineNumber="15" covered="true"></lineToCover>
    <lineToCover lineNumber="16" covered="true"></lineToCover>
    <lineToCover lineNumber="17" covered="true"></lineToCover>
    <lineToCover lineNumber="18" covered="true"></lineToCover>
    <lineToCover lineNumber="23" covered="true"></lineToCover>
    <lineToCover lineNumber="24" covered="true"></lineToCover>
    <lineToCover lineNumber="25" covered="true"></lineToCover>
    <lineToCover lineNumber="26" covered="true"></lineToCover>
    <lineToCover lineNumber="27" covered="true"></lineToCover>
    <lineToCover lineNumber="28" covered="true"></lineToCover>
    <lineToCover lineNumber="29" covered="true"></lineToCover>
    <lineToCover lineNumber="30" covered="true"></lineToCover>
    <lineToCover lineNumber="31" covered="true"></lineToCover>
    <lineToCover lineNumber="32" covered="true"></lineToCover>
    <lineToCover lineNumber="33" covered="true"></lineToCover>
    <lineToCover lineNumber="34" covered="true"></lineToCover>
    <lineToCover lineNumber="35" covered="false"></lineToCover>
    <lineToCover lineNumber="36" covered="false"></lineToCover>
    <lineToCover lineNumber="37" covered="false"></lineToCover>
    <lineToCover lineNumber="38" covered="false"></lineToCover>
    <lineToCover lineNumber="39" covered="false"></lineToCover>
    <lineToCover lineNumber="40" covered="false"></lineToCover>
    <lineToCover lineNumber="49" covered="true"></lineToCover>
    <lineToCover lineNumber="50" covered="true"></lineToCover>
    <lineToCover lineNumber="51" covered="true"></lineToCover>
    <lineToCover lineNumber="52" covered="true"></lineToCover>
  </file>
  <!--Cobertura for pipeline test-->
</coverage>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add best-effort coverage reports per test type jsoriano/elastic-package
1 participant
@mrodm