-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add E2E receiver/export correctness tests #652
Comments
I will be happy to help with this :) |
I will soon be submitting a PR for generating and managing "Golden Data". It will have the following components and process steps:
|
@kbrockhoff great, this will be a very useful addition. Please make smaller incremental PRs if possible to make reviewing easier. |
1. PICT tool input files for trace data and resulting output files from running pict from the command line. These are in `internal/goldendataset/testdata`. See: [Pairwise Independent Combinatorial Testing](https://github.com/microsoft/pict) 2. Generator for fully-populated OLTP ResourceSpans from PICT output files. This has all the intended functionality for this PR. It has no impact on other functionality so there should be no problem in merging to master. I will follow up with other PRs to complete full "Golden Dataset" testing functionality. **Link to tracking Issue:** #652 **Testing:** Unit tests exist for all functionality which has been coded so far **Documentation:** None yet. Will do so after golden dataset generation coding is complete.
I plan to write these tests to verify the API in the generalize-testbed branch. If @pmcollins has not started, you can assign the ticket to me. Or else I am happy to advise on how to write the tests using the generalized testbed. |
Either way works for me, @kbrockhoff . I was part way through a proof of concept for how to test for correctness: two pipelines, a pipeline under test and a test harness pipeline. The test harness pipeline has a processor that sends metrics to an exporter that is configured to talk to the pipeline under test, from which it is configured to also receive metrics. The same processor in the test harness pipeline compares the received metrics to what it sent. But maybe the Originally, I think we, including @tigrannajaryan, thought that maybe you (or someone) could work on the traces tests and I could work on the metrics tests (I'm more familiar with metrics). But maybe the way to go is for one of us to hold off until the other has an implementation. I'm happy to be the one to hold off since it looks like you have made significantly more progress than I have. |
I think since @kbrockhoff started the trace PICT generator it is best that he continues working on it and @pmcollins you can work on the similar capability and tests for metrics part. Pablo, you are right that it may be best to wait a bit until Kevin is done with testbed refactoring. Kevin, do you need more changes to the testbed after this PR is merged? |
I was planning to do all the refactoring in one PR. Still have a few improvements to make yet before it is ready for merging. |
Extracted out TestResultsSummary (in testbed/testbed/results.go), DataProvider (in testbed/testbed/data_provider.go), OtelcolRunner (in testbed/testbed/otelcol_runner.go), TestCaseValidator (in testbed/testbed/validator.go) interfaces with multiple implementations. Added tracing correctness tests in testbed/correctness using the testbed with different implementations of the 5 interfaces listed than what the perf tests use. **Link to tracking Issue:** Provides the support to cleanly implement #652, #1022, #1023, #1027, #1031 **Testing:** All existing testbed-driven tests still pass. Correctness tests run without any panics. Correctness tests are reporting a number of bugs with translations. **Documentation:** Godocs on all public methods.
Testbed changes have been merged to master. Correctness tests for traces have been completed as part of the PR. Work on metrics correctness tests can now proceed. |
1. PICT tool input files for trace data and resulting output files from running pict from the command line. These are in `internal/goldendataset/testdata`. See: [Pairwise Independent Combinatorial Testing](https://github.com/microsoft/pict) 2. Generator for fully-populated OLTP ResourceSpans from PICT output files. This has all the intended functionality for this PR. It has no impact on other functionality so there should be no problem in merging to master. I will follow up with other PRs to complete full "Golden Dataset" testing functionality. **Link to tracking Issue:** open-telemetry#652 **Testing:** Unit tests exist for all functionality which has been coded so far **Documentation:** None yet. Will do so after golden dataset generation coding is complete.
…telemetry#1062) Extracted out TestResultsSummary (in testbed/testbed/results.go), DataProvider (in testbed/testbed/data_provider.go), OtelcolRunner (in testbed/testbed/otelcol_runner.go), TestCaseValidator (in testbed/testbed/validator.go) interfaces with multiple implementations. Added tracing correctness tests in testbed/correctness using the testbed with different implementations of the 5 interfaces listed than what the perf tests use. **Link to tracking Issue:** Provides the support to cleanly implement open-telemetry#652, open-telemetry#1022, open-telemetry#1023, open-telemetry#1027, open-telemetry#1031 **Testing:** All existing testbed-driven tests still pass. Correctness tests run without any panics. Correctness tests are reporting a number of bugs with translations. **Documentation:** Godocs on all public methods.
* Move some content from correctness_test.go to utils.go This change makes these functions/types available from the metrics package, where they will be needed to address issue #652. * Add comments for exported types and fcns * Address PR comments * Fix lint
Closing this issue, correctness tests now exist. |
…t in all examples (open-telemetry#652) * fixed the image version of busybox to latest in all examples * bumped chart version after rebase
We currently have E2E tests that benchmark the performance of various formats.
We also need E2E tests that verify the correctness of the Collector operation as it receives and exports the data in various formats. Performance tests don't verify this today. We need separate tests that will send telemetry data to the Collector, covering all possible variety of such data and then verify that the Collector exports this data precisely as it is supposed to be represented in the configured export format.
The preference is to have a matrix test that verifies many receiver/exporter combinations and uses golden data sets for verification.
Possible approach:
ScenarioTraceTranslation
and also call it in Contrib to test receivers and exporters in Contrib. Enable contrib processors (e.g. k8s processor).The text was updated successfully, but these errors were encountered: