feat: create cache rollup cron task #857

joseph-sentry · 2024-11-06T20:06:33Z

this is a task that runs nightly and queues up cache test rollups tasks
based on the content of the last_rollup_date table.

basically if a rollup exists that was valid yesterday, we want to
recalculate the aggregates for the next day. Eventually the branch's
rollups become so out of date that we don't care about recalculating
the aggregates anymore

depends on: #821

this commit does 2 things: - creates the cache test rollups task which runs the test rollups aggregation SQL and updates the redis cache with the results in pyarrow ipc format for that specific branch and interval. For the default branch of the repo then that cache entry won't be expired else it will be expired after 3 days. - triggers this task in the test results finisher - adds tests for the above

this commit switches the cache test rollups task to using polars instead of pyarrow. It seems polars uses pyarrow internally anyways but we want to use polars in the api for additional filtering and ordering of the results we calculate and store in the cache task, so for parity with the api we use polars here as well.

we want to have an async task that will get the cached rollup aggregates from GCS and put them in redis for an hour. the reason this is async is because we don't want to spend too much time uploading to redis in the API call

…st-results

this is a task that runs nightly and queues up cache test rollups tasks based on the content of the last_rollup_date table. basically if a rollup exists that was valid yesterday, we want to recalculate the aggregates for the next day. Eventually the branch's rollups become so out of date that we don't care about recalculating the aggregates anymore

sentry-io · 2024-11-06T20:06:42Z

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: tasks/test_results_finisher.py

Function	Unhandled Issue
`process_impl_within_lock`	AttributeError: 'TestResultsFinisherTask' object has no attribute 'schedule_new_user_activated_task' ... `Event Count:` 1

_{Did you find this useful? React with a 👍 or 👎}

github-actions · 2024-11-06T20:06:50Z

This PR includes changes to shared. Please review them here: codecov/shared@83b3376...bf285c7

codecov · 2024-11-06T20:10:36Z

Codecov Report

Attention: Patch coverage is 98.61111% with 1 line in your changes missing coverage. Please review.

Project coverage is 98.01%. Comparing base (03c73da) to head (d5f9c17).
Report is 2 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
celery_config.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #857      +/-   ##
==========================================
- Coverage   98.06%   98.01%   -0.05%     
==========================================
  Files         444      446       +2     
  Lines       35474    35608     +134     
==========================================
+ Hits        34786    34902     +116     
- Misses        688      706      +18

Flag	Coverage Δ
integration	`41.99% <43.05%> (+0.05%)`	⬆️
unit	`90.88% <98.61%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

codecov-staging · 2024-11-06T20:10:36Z

Codecov Report

Attention: Patch coverage is 98.61111% with 1 line in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
celery_config.py	66.66%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

codecov-qa · 2024-11-06T20:10:45Z

❌ 12 Tests Failed:

Tests completed	Failed	Passed	Skipped
12	12	0	0

View the top 3 failed tests by shortest run time

 tasks.tests.integration.test_ghm_sync_plans

Stack Traces | 0s run time

No failure message available

 tasks.tests.integration.test_status_set_pending_task

Stack Traces | 0s run time

No failure message available

 tasks.tests.integration.test_timeseries_backfill

Stack Traces | 0s run time

No failure message available

To view more test analytics, go to the Test Analytics Dashboard
Got feedback? Let us know on Github

codecov-public-qa · 2024-11-06T20:10:53Z

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 12 tests with 12 failed, 0 passed and 0 skipped.

View the full list of failed tests

pytest

Class name:
Test name: services.tests.test_repository_service
```
No failure message available
```
Class name:
Test name: tasks.tests.integration.test_ai_pr_review
```
No failure message available
```
Class name:
Test name: tasks.tests.integration.test_ghm_sync_plans
```
No failure message available
```
Class name:
Test name: tasks.tests.integration.test_http_request_task
```
No failure message available
```
Class name:
Test name: tasks.tests.integration.test_notify_task
```
No failure message available
```
Class name:
Test name: tasks.tests.integration.test_send_email_task
```
No failure message available
```
Class name:
Test name: tasks.tests.integration.test_status_set_error_task
```
No failure message available
```
Class name:
Test name: tasks.tests.integration.test_status_set_pending_task
```
No failure message available
```
Class name:
Test name: tasks.tests.integration.test_sync_pull
```
No failure message available
```
Class name:
Test name: tasks.tests.integration.test_timeseries_backfill
```
No failure message available
```
Class name:
Test name: tasks.tests.integration.test_upload_e2e
```
No failure message available
```
Class name:
Test name: tasks.tests.unit
```
No failure message available
```

…cache-cron

github-actions · 2024-11-18T20:58:17Z

✅ All tests successful. No failed tests were found.

📣 Thoughts on this report? Let Codecov know! | Powered by Codecov

Swatinem · 2024-11-19T09:30:00Z

tasks/cache_test_rollups.py

@@ -163,6 +166,16 @@ def run_impl_within_lock(self, repoid, branch):
                )

                serialized_table = df.write_ipc(None)
+
+                storage_service.write_file("codecov", storage_key, serialized_table)


you already call write_file down below, so I’m pretty sure this is misplaced. also you hardcode the codecov bucket which is wrong.

Swatinem · 2024-11-19T09:34:06Z

tasks/cache_rollup_cron_task.py

+                self.app.tasks[cache_test_rollups_task_name].create_signature(
+                    args=None,
+                    kwargs=dict(repoid=repo.repoid, branch=branch, update_date=False),
+                ).apply_async()


Suggested change

self.app.tasks[cache_test_rollups_task_name].create_signature(

args=None,

kwargs=dict(repoid=repo.repoid, branch=branch, update_date=False),

).apply_async()

self.app.tasks[cache_test_rollups_task_name].s(repoid=repo.repoid, branch=branch, update_date=False).apply_async()

TBH, its really weird how celery names this function just s, but its the version of the function where you give the arguments just as you would, without explicitly spelling out the kwargs.

use celery s function instead of create_signature and remove unnecessary write_file call

joseph-sentry added 8 commits November 1, 2024 09:56

fix: test flag bridge creation in test results processing

5b0c7c1

feat: create cache_test_rollups_redis

5aa46a1

we want to have an async task that will get the cached rollup aggregates from GCS and put them in redis for an hour. the reason this is async is because we don't want to spend too much time uploading to redis in the API call

build: update shared

874d409

Merge branch 'main' of github.com:codecov/worker into joseph/cache-te…

003e853

…st-results

fix: add caching tasks to app

70dbe9c

joseph-sentry added 2 commits November 18, 2024 15:46

Merge branch 'main' of github.com:codecov/worker into joseph/nightly-…

34b3620

…cache-cron

test: fix tests

1e51a29

joseph-sentry marked this pull request as ready for review November 18, 2024 20:51

joseph-sentry requested a review from a team November 18, 2024 20:51

Swatinem reviewed Nov 19, 2024

View reviewed changes

fix: address feedback

f1ecd84

use celery s function instead of create_signature and remove unnecessary write_file call

Swatinem approved these changes Nov 19, 2024

View reviewed changes

joseph-sentry enabled auto-merge November 19, 2024 16:54

test: fix tests

d5f9c17

joseph-sentry added this pull request to the merge queue Nov 19, 2024

Merged via the queue into main with commit a3ca1d9 Nov 19, 2024
20 of 27 checks passed

joseph-sentry deleted the joseph/nightly-cache-cron branch November 19, 2024 22:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: create cache rollup cron task #857

feat: create cache rollup cron task #857

joseph-sentry commented Nov 6, 2024

sentry-io bot commented Nov 6, 2024

github-actions bot commented Nov 6, 2024

codecov bot commented Nov 6, 2024 •

edited

Loading

codecov-staging bot commented Nov 6, 2024 •

edited by codecov-notifications bot

Loading

codecov-qa bot commented Nov 6, 2024

codecov-public-qa bot commented Nov 6, 2024

pytest

github-actions bot commented Nov 18, 2024

Swatinem Nov 19, 2024

Swatinem Nov 19, 2024

feat: create cache rollup cron task #857

feat: create cache rollup cron task #857

Conversation

joseph-sentry commented Nov 6, 2024

sentry-io bot commented Nov 6, 2024

🔍 Existing Issues For Review

github-actions bot commented Nov 6, 2024

codecov bot commented Nov 6, 2024 • edited Loading

Codecov Report

codecov-staging bot commented Nov 6, 2024 • edited by codecov-notifications bot Loading

Codecov Report

codecov-qa bot commented Nov 6, 2024

❌ 12 Tests Failed:

codecov-public-qa bot commented Nov 6, 2024

❌ Failed Test Results:

pytest

github-actions bot commented Nov 18, 2024

Swatinem Nov 19, 2024

Choose a reason for hiding this comment

Swatinem Nov 19, 2024

Choose a reason for hiding this comment

codecov bot commented Nov 6, 2024 •

edited

Loading

codecov-staging bot commented Nov 6, 2024 •

edited by codecov-notifications bot

Loading