Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pkg/stanza] Flaky test TestDeleteAfterRead_SkipPartials #27845

Closed
dmitryax opened this issue Oct 18, 2023 · 4 comments · Fixed by #29597
Closed

[pkg/stanza] Flaky test TestDeleteAfterRead_SkipPartials #27845

dmitryax opened this issue Oct 18, 2023 · 4 comments · Fixed by #29597
Labels
bug Something isn't working flaky test a test is flaky pkg/stanza priority:p2 Medium

Comments

@dmitryax
Copy link
Member

dmitryax commented Oct 18, 2023 •
edited
Loading

https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/6562693797/job/17826173436?pr=27825

--- FAIL: TestDeleteAfterRead_SkipPartials (10.36s)
    file_test.go:1627: 
        	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/file_test.go:1627
        	Error:      	file "/tmp/TestDeleteAfterRead_SkipPartials2080466458/001/1867497531" exists
        	Test:       	TestDeleteAfterRead_SkipPartials
FAIL
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer	coverage: [88](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/6562693797/job/17826173436?pr=27825#step:9:89).6% of statements
FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer	43.800s
@dmitryax dmitryax added bug Something isn't working priority:p2 Medium flaky test a test is flaky pkg/stanza labels Oct 18, 2023
@github-actions
Copy link
Contributor

Pinging code owners for pkg/stanza: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot mentioned this issue Oct 24, 2023
Closed
@pjanotti
Copy link
Contributor

Another hit on the same test
https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/6922266602/job/18828797459?pr=29343#step:9:97

--- FAIL: TestDeleteAfterRead_SkipPartials (9.44s)
    file_test.go:1556: 
        	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/file_test.go:1556
        	Error:      	file "/tmp/TestDeleteAfterRead_SkipPartials3022013525/001/425630[104](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/6922266602/job/18828797459?pr=29343#step:9:105)8" exists
        	Test:       	TestDeleteAfterRead_SkipPartials
FAIL
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer	coverage: 87.7% of statements
FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer	44.366s

@pjanotti pjanotti mentioned this issue Nov 19, 2023
Merged
@sakulali
Copy link
Contributor

+1: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/7015259719/job/19084304995?pr=29530

@djaglowski djaglowski mentioned this issue Nov 30, 2023
Merged
djaglowski added a commit that referenced this issue Dec 1, 2023
@djaglowski
[pkg/stanza] Fix flaky TestDeleteAfterRead_SkipPartials (#29597)
03ad502 
Fixes #27845 

The failure mechanism was very subtle. The test waits for the contents
of the short file to be emitted before canceling the context which is
used in the file reader. However, the reader may require one additional
scan to determine that it's reached the end of the file, and therefore
in a tiny proportion of cases, the context cancelation could actually
trigger the reader to return before it had determined it should delete
the file. The solution involves validating file deletion before
canceling the context. I was able to reproduce failure locally only
about 1/3000 times, but with this change was able to run 30,000 times
without a failure. The PR also cleans up a few related parts of the
code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working flaky test a test is flaky pkg/stanza priority:p2 Medium
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[pkg/stanza] Fix flaky TestDeleteAfterRead_SkipPartials djaglowski/opentelemetry-collector-contrib
3 participants
@sakulali @pjanotti @dmitryax