Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid copying raw upload for parallel verification #704

Merged
merged 2 commits into from
Sep 16, 2024

Conversation

Swatinem
Copy link
Contributor

Instead of creating a copy of a raw upload for consumption in the parallel verification task, this will only schedule parallel verification in case the uploads are not being deleted after processing.

That way, the uploads are guaranteed to still be there for verification, so no copy is needed.

Instead of creating a copy of a raw upload for consumption in the parallel verification task,
this will only schedule parallel verification in case the uploads are not being deleted after processing.

That way, the uploads are guaranteed to still be there for verification, so no copy is needed.
@Swatinem Swatinem requested a review from a team September 12, 2024 10:17
@Swatinem Swatinem self-assigned this Sep 12, 2024
@codecov-notifications
Copy link

codecov-notifications bot commented Sep 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

✅ All tests successful. No failed tests found.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #704      +/-   ##
==========================================
- Coverage   98.09%   98.09%   -0.01%     
==========================================
  Files         434      434              
  Lines       36618    36613       -5     
==========================================
- Hits        35922    35917       -5     
  Misses        696      696              
Flag Coverage Δ
integration 98.09% <100.00%> (-0.01%) ⬇️
latest-uploader-overall 98.09% <100.00%> (-0.01%) ⬇️
unit 98.09% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 96.01% <100.00%> (-0.01%) ⬇️
OutsideTasks 98.10% <100.00%> (-0.01%) ⬇️
Files Coverage Δ
helpers/reports.py 100.00% <100.00%> (ø)
services/report/__init__.py 97.36% <100.00%> (-0.06%) ⬇️
services/yaml/reader.py 100.00% <100.00%> (ø)
tasks/tests/unit/test_upload_processing_task.py 100.00% <100.00%> (ø)
tasks/tests/unit/test_upload_task.py 99.54% <ø> (ø)
tasks/upload.py 96.35% <100.00%> (+0.02%) ⬆️
tasks/upload_processor.py 99.45% <100.00%> (-0.02%) ⬇️

@codecov-qa
Copy link

codecov-qa bot commented Sep 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.09%. Comparing base (7ebdf9f) to head (1d25d4e).
Report is 3 commits behind head on main.

✅ All tests successful. No failed tests found.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #704      +/-   ##
==========================================
- Coverage   98.09%   98.09%   -0.01%     
==========================================
  Files         434      434              
  Lines       36618    36613       -5     
==========================================
- Hits        35922    35917       -5     
  Misses        696      696              
Flag Coverage Δ
integration 98.09% <100.00%> (-0.01%) ⬇️
latest-uploader-overall 98.09% <100.00%> (-0.01%) ⬇️
unit 98.09% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 96.01% <100.00%> (-0.01%) ⬇️
OutsideTasks 98.10% <100.00%> (-0.01%) ⬇️
Files with missing lines Coverage Δ
helpers/reports.py 100.00% <100.00%> (ø)
services/report/__init__.py 97.36% <100.00%> (-0.06%) ⬇️
services/yaml/reader.py 100.00% <100.00%> (ø)
tasks/tests/unit/test_upload_processing_task.py 100.00% <100.00%> (ø)
tasks/tests/unit/test_upload_task.py 99.54% <ø> (ø)
tasks/upload.py 96.35% <100.00%> (+0.02%) ⬆️
tasks/upload_processor.py 99.45% <100.00%> (-0.02%) ⬇️

Copy link

codecov-public-qa bot commented Sep 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.09%. Comparing base (7ebdf9f) to head (1d25d4e).

✅ All tests successful. No failed tests found.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #704      +/-   ##
==========================================
- Coverage   98.09%   98.09%   -0.01%     
==========================================
  Files         434      434              
  Lines       36618    36613       -5     
==========================================
- Hits        35922    35917       -5     
  Misses        696      696              
Flag Coverage Δ
integration 98.09% <100.00%> (-0.01%) ⬇️
latest-uploader-overall 98.09% <100.00%> (-0.01%) ⬇️
unit 98.09% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 96.01% <100.00%> (-0.01%) ⬇️
OutsideTasks 98.10% <100.00%> (-0.01%) ⬇️
Files Coverage Δ
helpers/reports.py 100.00% <100.00%> (ø)
services/report/__init__.py 97.36% <100.00%> (-0.06%) ⬇️
services/yaml/reader.py 100.00% <100.00%> (ø)
tasks/tests/unit/test_upload_processing_task.py 100.00% <100.00%> (ø)
tasks/tests/unit/test_upload_task.py 99.54% <ø> (ø)
tasks/upload.py 96.35% <100.00%> (+0.02%) ⬆️
tasks/upload_processor.py 99.45% <100.00%> (-0.02%) ⬇️

Copy link

codecov bot commented Sep 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.14%. Comparing base (7ebdf9f) to head (1d25d4e).
Report is 3 commits behind head on main.

Changes have been made to critical files, which contain lines commonly executed in production. Learn more

✅ All tests successful. No failed tests found.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #704      +/-   ##
==========================================
- Coverage   98.14%   98.14%   -0.01%     
==========================================
  Files         475      475              
  Lines       37973    37968       -5     
==========================================
- Hits        37268    37263       -5     
  Misses        705      705              
Flag Coverage Δ
integration 98.09% <100.00%> (-0.01%) ⬇️
latest-uploader-overall 98.09% <100.00%> (-0.01%) ⬇️
unit 98.09% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 96.12% <100.00%> (-0.01%) ⬇️
OutsideTasks 98.10% <100.00%> (-0.01%) ⬇️
Files with missing lines Coverage Δ
helpers/reports.py 100.00% <100.00%> (ø)
services/report/__init__.py Critical 97.36% <100.00%> (-0.06%) ⬇️
services/yaml/reader.py 100.00% <100.00%> (ø)
tasks/tests/unit/test_upload_processing_task.py 100.00% <100.00%> (ø)
tasks/tests/unit/test_upload_task.py 99.54% <ø> (ø)
tasks/upload.py Critical 97.69% <100.00%> (+0.01%) ⬆️
tasks/upload_processor.py 99.45% <100.00%> (-0.02%) ⬇️
Related Entrypoints
run/app.tasks.upload.Upload
run/app.tasks.status.SetError
run/app.tasks.notify.Notify
run/app.tasks.profiling.normalizer
run/app.tasks.pulls.Sync
run/app.tasks.upload.UploadFinisher
run/app.tasks.upload.UploadProcessor
run/app.tasks.upload.PreProcessUpload
run/app.tasks.bundle_analysis.BundleAnalysisNotify
run/app.tasks.test_results.TestResultsProcessor

@Swatinem Swatinem force-pushed the swatinem/no-copy-upload branch from a9a3470 to e94e4d0 Compare September 12, 2024 10:55
@Swatinem Swatinem force-pushed the swatinem/no-copy-upload branch from e94e4d0 to 1d25d4e Compare September 12, 2024 11:10
Copy link
Contributor

@giovanni-guidini giovanni-guidini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Just pointing out that processing might change the format of the raw report to a "readable format" (legacy) if it was uploaded as v1 format. I don't think this matters much, but it might affect performance metrics if we are collecting those.

@Swatinem
Copy link
Contributor Author

Indeed. We might end up seeing some more Sentry logs about wrong report format, though I changed those to be compatible with both formats anyway.

@Swatinem Swatinem added this pull request to the merge queue Sep 16, 2024
Merged via the queue into main with commit 10c910a Sep 16, 2024
26 of 27 checks passed
@Swatinem Swatinem deleted the swatinem/no-copy-upload branch September 16, 2024 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants