Store v2 reports in GCS instead of Redis #960

Swatinem · 2024-11-05T13:43:57Z

Worker has some dedicated code to move an uploaded report from Redis over to GCS. Instead of having that code, we can just store the report into GCS in the first place.

Additionally, this PR also cleans up the services.archive, replacing it fully with the shared.api_archive.archive.

It also forwards all the task arguments directly to the Upload task, in addition to storing it in Redis, to be forward compatible to some more planned upload processing improvements.

Depends on codecov/shared#415

codecov · 2024-11-05T13:55:49Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.06%. Comparing base (9f3c523) to head (fc82601).
Report is 1 commits behind head on main.

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #960      +/-   ##
==========================================
- Coverage   96.06%   96.06%   -0.01%     
==========================================
  Files         828      828              
  Lines       19279    19275       -4     
==========================================
- Hits        18520    18516       -4     
  Misses        759      759

Flag	Coverage Δ
unit	`92.33% <100.00%> (-0.01%)`	⬇️
unit-latest-uploader	`92.33% <100.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

codecov-notifications · 2024-11-05T13:55:52Z

Codecov Report

All modified and coverable lines are covered by tests ✅

✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

codecov-qa · 2024-11-05T13:55:56Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
2652	2	2650	6

View the top 2 failed tests by shortest run time

upload/tests/test_upload.py::UploadHandlerRouteTest::test_successful_upload_v2_slash

Stack Traces | 0.05s run time

self = &lt;MagicMock name='write_file' id='139671276570912'&gt;
args = ('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
kwargs = {}
expected = call('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
actual = call('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
_error_message = &lt;function NonCallableMock.assert_called_with.&lt;locals&gt;._error_message at 0x7f077c6e7b00&gt;
cause = None

    def assert_called_with(self, /, *args, **kwargs):
        """assert that the last call was made with the specified arguments.
    
        Raises an AssertionError if the args and keyword args passed in are
        different to the last call to the mock."""
        if self.call_args is None:
            expected = self._format_mock_call_signature(args, kwargs)
            actual = 'not called.'
            error_message = ('expected call not found.\nExpected: %s\n  Actual: %s'
                    % (expected, actual))
            raise AssertionError(error_message)
    
        def _error_message():
            msg = self._format_mock_failure_message(args, kwargs)
            return msg
        expected = self._call_matcher(_Call((args, kwargs), two=True))
        actual = self._call_matcher(self.call_args)
        if actual != expected:
            cause = expected if isinstance(expected, Exception) else None
&gt;           raise AssertionError(_error_message()) from cause
E           AssertionError: expected call not found.
E           Expected: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
E             Actual: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)

.../local/lib/python3.12/unittest/mock.py:949: AssertionError

During handling of the above exception, another exception occurred:

self = &lt;upload.tests.test_upload.UploadHandlerRouteTest testMethod=test_successful_upload_v2_slash&gt;
mock_repo_provider_service = &lt;MagicMock name='get_adapter' id='139670663700240'&gt;
mock_dispatch_upload = &lt;MagicMock name='dispatch_upload_task' id='139670137903856'&gt;
mock_uuid4 = &lt;MagicMock name='uuid4' id='139671605272432'&gt;
mock_get_redis = &lt;MagicMock name='get_redis_connection' id='139670130925712'&gt;
mock_write_file = &lt;MagicMock name='write_file' id='139671276570912'&gt;

    @patch("shared.api_archive.archive.ArchiveService.write_file")
    @patch("upload.views.legacy.get_redis_connection")
    @patch("upload.views.legacy.uuid4")
    @patch("upload.views.legacy.dispatch_upload_task")
    @patch("services.repo_providers.RepoProviderService.get_adapter")
    @override_settings(CODECOV_DASHBOARD_URL="https://app.codecov.io")
    def test_successful_upload_v2_slash(
        self,
        mock_repo_provider_service,
        mock_dispatch_upload,
        mock_uuid4,
        mock_get_redis,
        mock_write_file,
    ):
        class MockRepoProviderAdapter:
            async def get_commit(self, commit, token):
                return {"message": "This is not a merge commit"}
    
        mock_get_redis.return_value = MockRedis()
        mock_repo_provider_service.return_value = MockRepoProviderAdapter()
        mock_uuid4.return_value = (
            "dec1f00b-1883-40d0-afd6-6dcb876510be"  # this will be the reportid
        )
    
        query_params = {
            "commit": "b521e55aef79b101f48e2544837ca99a7fa3bf6b",
            "token": "a03e5d02-9495-4413-b0d8-05651bb2e842",
            "pr": "456",
            "branch": "",
            "flags": "",
            "build_url": "",
            "package": "",
        }
    
        response = self._post_slash(
            kwargs={"version": "v2"}, query=query_params, data="coverage report"
        )
    
        assert response.status_code == 200
    
        headers = response.headers
    
        assert headers["access-control-allow-origin"] == "*"
        assert (
            headers["access-control-allow-headers"]
            == "Origin, Content-Type, Accept, X-User-Agent"
        )
        assert headers["content-type"] != "text/plain"
    
        archive_service = ArchiveService(self.repo)
        datetime = timezone.now().strftime("%Y-%m-%d")
        repo_hash = archive_service.get_archive_hash(self.repo)
        expected_url = f"v4/raw/{datetime}/{repo_hash}/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt"
    
&gt;       mock_write_file.assert_called_with(expected_url, ANY, ANY)
E       AssertionError: expected call not found.
E       Expected: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
E         Actual: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
E       
E       pytest introspection follows:
E       
E       Args:
E       assert ('v4/raw/2024...erage report') == ('v4/raw/2024... &lt;ANY&gt;, &lt;ANY&gt;)
E         
E         Right contains one more item: &lt;ANY&gt;
E         Use -v to get more diff
E       Kwargs:
E       assert {'is_already_gzipped': False} == {}
E         
E         Left contains 1 more item:
E         {'is_already_gzipped': False}
E         Use -v to get more diff

upload/tests/test_upload.py:1100: AssertionError

upload/tests/test_upload.py::UploadHandlerRouteTest::test_successful_upload_v2

Stack Traces | 0.059s run time

self = &lt;MagicMock name='write_file' id='139670940359360'&gt;
args = ('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
kwargs = {}
expected = call('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
actual = call('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
_error_message = &lt;function NonCallableMock.assert_called_with.&lt;locals&gt;._error_message at 0x7f07acefd760&gt;
cause = None

    def assert_called_with(self, /, *args, **kwargs):
        """assert that the last call was made with the specified arguments.
    
        Raises an AssertionError if the args and keyword args passed in are
        different to the last call to the mock."""
        if self.call_args is None:
            expected = self._format_mock_call_signature(args, kwargs)
            actual = 'not called.'
            error_message = ('expected call not found.\nExpected: %s\n  Actual: %s'
                    % (expected, actual))
            raise AssertionError(error_message)
    
        def _error_message():
            msg = self._format_mock_failure_message(args, kwargs)
            return msg
        expected = self._call_matcher(_Call((args, kwargs), two=True))
        actual = self._call_matcher(self.call_args)
        if actual != expected:
            cause = expected if isinstance(expected, Exception) else None
&gt;           raise AssertionError(_error_message()) from cause
E           AssertionError: expected call not found.
E           Expected: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
E             Actual: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)

.../local/lib/python3.12/unittest/mock.py:949: AssertionError

During handling of the above exception, another exception occurred:

self = &lt;upload.tests.test_upload.UploadHandlerRouteTest testMethod=test_successful_upload_v2&gt;
mock_repo_provider_service = &lt;MagicMock name='get_adapter' id='139670663059200'&gt;
mock_dispatch_upload = &lt;MagicMock name='dispatch_upload_task' id='139670938203056'&gt;
mock_uuid4 = &lt;MagicMock name='uuid4' id='139671601606784'&gt;
mock_get_redis = &lt;MagicMock name='get_redis_connection' id='139670940003040'&gt;
mock_write_file = &lt;MagicMock name='write_file' id='139670940359360'&gt;

    @patch("shared.api_archive.archive.ArchiveService.write_file")
    @patch("upload.views.legacy.get_redis_connection")
    @patch("upload.views.legacy.uuid4")
    @patch("upload.views.legacy.dispatch_upload_task")
    @patch("services.repo_providers.RepoProviderService.get_adapter")
    @override_settings(CODECOV_DASHBOARD_URL="https://app.codecov.io")
    def test_successful_upload_v2(
        self,
        mock_repo_provider_service,
        mock_dispatch_upload,
        mock_uuid4,
        mock_get_redis,
        mock_write_file,
    ):
        class MockRepoProviderAdapter:
            async def get_commit(self, commit, token):
                return {"message": "This is not a merge commit"}
    
        mock_get_redis.return_value = MockRedis()
        mock_repo_provider_service.return_value = MockRepoProviderAdapter()
        mock_uuid4.return_value = (
            "dec1f00b-1883-40d0-afd6-6dcb876510be"  # this will be the reportid
        )
    
        query_params = {
            "commit": "b521e55aef79b101f48e2544837ca99a7fa3bf6b",
            "token": "a03e5d02-9495-4413-b0d8-05651bb2e842",
            "pr": "456",
            "branch": "",
            "flags": "",
            "build_url": "",
            "package": "",
        }
    
        response = self._post(
            kwargs={"version": "v2"}, query=query_params, data="coverage report"
        )
    
        assert response.status_code == 200
    
        headers = response.headers
    
        assert headers["access-control-allow-origin"] == "*"
        assert (
            headers["access-control-allow-headers"]
            == "Origin, Content-Type, Accept, X-User-Agent"
        )
        assert headers["content-type"] != "text/plain"
    
        archive_service = ArchiveService(self.repo)
        datetime = timezone.now().strftime("%Y-%m-%d")
        repo_hash = archive_service.get_archive_hash(self.repo)
        expected_url = f"v4/raw/{datetime}/{repo_hash}/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt"
    
&gt;       mock_write_file.assert_called_with(expected_url, ANY, ANY)
E       AssertionError: expected call not found.
E       Expected: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
E         Actual: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
E       
E       pytest introspection follows:
E       
E       Args:
E       assert ('v4/raw/2024...erage report') == ('v4/raw/2024... &lt;ANY&gt;, &lt;ANY&gt;)
E         
E         Right contains one more item: &lt;ANY&gt;
E         Use -v to get more diff
E       Kwargs:
E       assert {'is_already_gzipped': False} == {}
E         
E         Left contains 1 more item:
E         {'is_already_gzipped': False}
E         Use -v to get more diff

upload/tests/test_upload.py:1019: AssertionError

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

github-actions · 2024-11-06T10:49:37Z

This PR includes changes to shared. Please review them here: codecov/shared@ebafd80...ce68890

codecov-public-qa · 2024-11-06T11:03:37Z

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 2658 tests with 2 failed, 2650 passed and 6 skipped.

View the full list of failed tests

pytest

Class name: upload.tests.test_upload.UploadHandlerRouteTest
Test name: test_successful_upload_v2

self = <MagicMock name='write_file' id='139670940359360'>
args = ('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
kwargs = {}
expected = call('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
actual = call('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
_error_message = <function NonCallableMock.assert_called_with.<locals>._error_message at 0x7f07acefd760>
cause = None

    def assert_called_with(self, /, *args, **kwargs):
        """assert that the last call was made with the specified arguments.
    
        Raises an AssertionError if the args and keyword args passed in are
        different to the last call to the mock."""
        if self.call_args is None:
            expected = self._format_mock_call_signature(args, kwargs)
            actual = 'not called.'
            error_message = ('expected call not found.\nExpected: %s\n  Actual: %s'
                    % (expected, actual))
            raise AssertionError(error_message)
    
        def _error_message():
            msg = self._format_mock_failure_message(args, kwargs)
            return msg
        expected = self._call_matcher(_Call((args, kwargs), two=True))
        actual = self._call_matcher(self.call_args)
        if actual != expected:
            cause = expected if isinstance(expected, Exception) else None
>           raise AssertionError(_error_message()) from cause
E           AssertionError: expected call not found.
E           Expected: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
E             Actual: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)

.../local/lib/python3.12/unittest/mock.py:949: AssertionError

During handling of the above exception, another exception occurred:

self = <upload.tests.test_upload.UploadHandlerRouteTest testMethod=test_successful_upload_v2>
mock_repo_provider_service = <MagicMock name='get_adapter' id='139670663059200'>
mock_dispatch_upload = <MagicMock name='dispatch_upload_task' id='139670938203056'>
mock_uuid4 = <MagicMock name='uuid4' id='139671601606784'>
mock_get_redis = <MagicMock name='get_redis_connection' id='139670940003040'>
mock_write_file = <MagicMock name='write_file' id='139670940359360'>

    @patch("shared.api_archive.archive.ArchiveService.write_file")
    @patch("upload.views.legacy.get_redis_connection")
    @patch("upload.views.legacy.uuid4")
    @patch("upload.views.legacy.dispatch_upload_task")
    @patch("services.repo_providers.RepoProviderService.get_adapter")
    @override_settings(CODECOV_DASHBOARD_URL="https://app.codecov.io")
    def test_successful_upload_v2(
        self,
        mock_repo_provider_service,
        mock_dispatch_upload,
        mock_uuid4,
        mock_get_redis,
        mock_write_file,
    ):
        class MockRepoProviderAdapter:
            async def get_commit(self, commit, token):
                return {"message": "This is not a merge commit"}
    
        mock_get_redis.return_value = MockRedis()
        mock_repo_provider_service.return_value = MockRepoProviderAdapter()
        mock_uuid4.return_value = (
            "dec1f00b-1883-40d0-afd6-6dcb876510be"  # this will be the reportid
        )
    
        query_params = {
            "commit": "b521e55aef79b101f48e2544837ca99a7fa3bf6b",
            "token": "a03e5d02-9495-4413-b0d8-05651bb2e842",
            "pr": "456",
            "branch": "",
            "flags": "",
            "build_url": "",
            "package": "",
        }
    
        response = self._post(
            kwargs={"version": "v2"}, query=query_params, data="coverage report"
        )
    
        assert response.status_code == 200
    
        headers = response.headers
    
        assert headers["access-control-allow-origin"] == "*"
        assert (
            headers["access-control-allow-headers"]
            == "Origin, Content-Type, Accept, X-User-Agent"
        )
        assert headers["content-type"] != "text/plain"
    
        archive_service = ArchiveService(self.repo)
        datetime = timezone.now().strftime("%Y-%m-%d")
        repo_hash = archive_service.get_archive_hash(self.repo)
        expected_url = f"v4/raw/{datetime}/{repo_hash}/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt"
    
>       mock_write_file.assert_called_with(expected_url, ANY, ANY)
E       AssertionError: expected call not found.
E       Expected: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
E         Actual: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
E       
E       pytest introspection follows:
E       
E       Args:
E       assert ('v4/raw/2024...erage report') == ('v4/raw/2024... <ANY>, <ANY>)
E         
E         Right contains one more item: <ANY>
E         Use -v to get more diff
E       Kwargs:
E       assert {'is_already_gzipped': False} == {}
E         
E         Left contains 1 more item:
E         {'is_already_gzipped': False}
E         Use -v to get more diff

upload/tests/test_upload.py:1019: AssertionError

Class name: upload.tests.test_upload.UploadHandlerRouteTest
Test name: test_successful_upload_v2_slash

self = <MagicMock name='write_file' id='139671276570912'>
args = ('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
kwargs = {}
expected = call('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
actual = call('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
_error_message = <function NonCallableMock.assert_called_with.<locals>._error_message at 0x7f077c6e7b00>
cause = None

    def assert_called_with(self, /, *args, **kwargs):
        """assert that the last call was made with the specified arguments.
    
        Raises an AssertionError if the args and keyword args passed in are
        different to the last call to the mock."""
        if self.call_args is None:
            expected = self._format_mock_call_signature(args, kwargs)
            actual = 'not called.'
            error_message = ('expected call not found.\nExpected: %s\n  Actual: %s'
                    % (expected, actual))
            raise AssertionError(error_message)
    
        def _error_message():
            msg = self._format_mock_failure_message(args, kwargs)
            return msg
        expected = self._call_matcher(_Call((args, kwargs), two=True))
        actual = self._call_matcher(self.call_args)
        if actual != expected:
            cause = expected if isinstance(expected, Exception) else None
>           raise AssertionError(_error_message()) from cause
E           AssertionError: expected call not found.
E           Expected: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
E             Actual: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)

.../local/lib/python3.12/unittest/mock.py:949: AssertionError

During handling of the above exception, another exception occurred:

self = <upload.tests.test_upload.UploadHandlerRouteTest testMethod=test_successful_upload_v2_slash>
mock_repo_provider_service = <MagicMock name='get_adapter' id='139670663700240'>
mock_dispatch_upload = <MagicMock name='dispatch_upload_task' id='139670137903856'>
mock_uuid4 = <MagicMock name='uuid4' id='139671605272432'>
mock_get_redis = <MagicMock name='get_redis_connection' id='139670130925712'>
mock_write_file = <MagicMock name='write_file' id='139671276570912'>

    @patch("shared.api_archive.archive.ArchiveService.write_file")
    @patch("upload.views.legacy.get_redis_connection")
    @patch("upload.views.legacy.uuid4")
    @patch("upload.views.legacy.dispatch_upload_task")
    @patch("services.repo_providers.RepoProviderService.get_adapter")
    @override_settings(CODECOV_DASHBOARD_URL="https://app.codecov.io")
    def test_successful_upload_v2_slash(
        self,
        mock_repo_provider_service,
        mock_dispatch_upload,
        mock_uuid4,
        mock_get_redis,
        mock_write_file,
    ):
        class MockRepoProviderAdapter:
            async def get_commit(self, commit, token):
                return {"message": "This is not a merge commit"}
    
        mock_get_redis.return_value = MockRedis()
        mock_repo_provider_service.return_value = MockRepoProviderAdapter()
        mock_uuid4.return_value = (
            "dec1f00b-1883-40d0-afd6-6dcb876510be"  # this will be the reportid
        )
    
        query_params = {
            "commit": "b521e55aef79b101f48e2544837ca99a7fa3bf6b",
            "token": "a03e5d02-9495-4413-b0d8-05651bb2e842",
            "pr": "456",
            "branch": "",
            "flags": "",
            "build_url": "",
            "package": "",
        }
    
        response = self._post_slash(
            kwargs={"version": "v2"}, query=query_params, data="coverage report"
        )
    
        assert response.status_code == 200
    
        headers = response.headers
    
        assert headers["access-control-allow-origin"] == "*"
        assert (
            headers["access-control-allow-headers"]
            == "Origin, Content-Type, Accept, X-User-Agent"
        )
        assert headers["content-type"] != "text/plain"
    
        archive_service = ArchiveService(self.repo)
        datetime = timezone.now().strftime("%Y-%m-%d")
        repo_hash = archive_service.get_archive_hash(self.repo)
        expected_url = f"v4/raw/{datetime}/{repo_hash}/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt"
    
>       mock_write_file.assert_called_with(expected_url, ANY, ANY)
E       AssertionError: expected call not found.
E       Expected: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
E         Actual: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
E       
E       pytest introspection follows:
E       
E       Args:
E       assert ('v4/raw/2024...erage report') == ('v4/raw/2024... <ANY>, <ANY>)
E         
E         Right contains one more item: <ANY>
E         Use -v to get more diff
E       Kwargs:
E       assert {'is_already_gzipped': False} == {}
E         
E         Left contains 1 more item:
E         {'is_already_gzipped': False}
E         Use -v to get more diff

upload/tests/test_upload.py:1100: AssertionError

matt-codecov

in future PRs could you put the actual logical changes in their own commit(s) so they're easier to find/review?

does worker already know that the uploads may no longer be in redis? or does that PR still need to be written

it doesn't look like the newer endpoint saves the report to redis in the first place

upload/views/legacy.py

Swatinem · 2024-11-07T08:40:33Z

in future PRs could you put the actual logical changes in their own commit(s) so they're easier to find/review?

Yes, I will try :-) One thing kind of led to the other, and I ended up just completely remove the duplicated ArchiveService/Storage in API, rather switching over to the one in shared.

does worker already know that the uploads may no longer be in redis?

Yes, the worker has backwards compatibility code for:

Move the upload from redis to GCS based on redis_key, moving: https://github.com/codecov/worker/blob/d369849095133aaf11c77d0afb8b4afd0cd55481/tasks/upload.py#L190-L202
Otherwise, it will use the url key to find an already stored upload.
It will then also optionally create the Upload object based on the upload_id: https://github.com/codecov/worker/blob/d369849095133aaf11c77d0afb8b4afd0cd55481/tasks/upload.py#L499-L503

requirements.in

adrian-codecov · 2024-11-27T17:41:12Z

services/task/task.py

@@ -134,6 +134,7 @@ def upload_signature(
        commitid,
        report_type=None,
        report_code=None,
+        arguments=None,


Do we use these in Worker?

Not yet, but I would like to make the task workflow simpler and make the batching more explicit.

codecov/engineering-team#2618 explains some of my ideas I am working towards.

In particular, I would love to get rid of the Upload -> N * UploadProcessor -> UploadFinisher fanout / map/reduce to a different strategy. Passing the arguments explicitly is a first step towards that.

services/tests/test_archive.py

Swatinem · 2024-11-28T09:58:49Z

I rebased this on top of #1006, so the diff should be smaller and focus only on the relevant parts.

adrian-codecov

Approved, so this should only be merged after #1006 is merged and you update to that latest Shared yeah? (also approved)

Worker has some dedicated code to move an uploaded report from Redis over to GCS. Instead of having that code, we can just store the report into GCS in the first place. It also forwards all the task arguments directly to the `Upload` task, in addition to storing it in Redis, to be forward compatible to some more planned upload processing improvements.

github-actions · 2024-11-29T09:00:58Z

✅ All tests successful. No failed tests were found.

📣 Thoughts on this report? Let Codecov know! | Powered by Codecov

Swatinem requested a review from adrian-codecov November 5, 2024 13:43

Swatinem self-assigned this Nov 5, 2024

Swatinem requested review from a team as code owners November 5, 2024 13:43

Swatinem force-pushed the swatinem/v2-store branch from 53e4286 to 09b0993 Compare November 6, 2024 10:49

Swatinem force-pushed the swatinem/v2-store branch 2 times, most recently from 38d2647 to 782fb8a Compare November 6, 2024 11:51

matt-codecov reviewed Nov 6, 2024

View reviewed changes

upload/views/legacy.py Outdated Show resolved Hide resolved

Swatinem mentioned this pull request Nov 26, 2024

feat: support zstd compression in miniostorage codecov/shared#405

Merged

adrian-codecov reviewed Nov 27, 2024

View reviewed changes

requirements.in Outdated Show resolved Hide resolved

adrian-codecov reviewed Nov 27, 2024

View reviewed changes

services/tests/test_archive.py Outdated Show resolved Hide resolved

Swatinem mentioned this pull request Nov 28, 2024

Fully replace services.archive by shared #1006

Merged

Swatinem force-pushed the swatinem/v2-store branch from 782fb8a to 71d164a Compare November 28, 2024 09:57

Swatinem changed the base branch from main to swatinem/shared-archive November 28, 2024 09:57

Swatinem force-pushed the swatinem/shared-archive branch from 74abbad to a2e3d61 Compare November 28, 2024 10:01

adrian-codecov approved these changes Nov 28, 2024

View reviewed changes

Base automatically changed from swatinem/shared-archive to main November 29, 2024 08:25

Swatinem force-pushed the swatinem/v2-store branch from 71d164a to fc82601 Compare November 29, 2024 08:49

Swatinem added this pull request to the merge queue Dec 2, 2024

Merged via the queue into main with commit 022c44b Dec 2, 2024
18 of 19 checks passed

Swatinem deleted the swatinem/v2-store branch December 2, 2024 08:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store v2 reports in GCS instead of Redis #960

Store v2 reports in GCS instead of Redis #960

Swatinem commented Nov 5, 2024 •

edited

Loading

codecov bot commented Nov 5, 2024 •

edited

Loading

codecov-notifications bot commented Nov 5, 2024 •

edited

Loading

codecov-qa bot commented Nov 5, 2024 •

edited

Loading

github-actions bot commented Nov 6, 2024

codecov-public-qa bot commented Nov 6, 2024 •

edited

Loading

pytest

matt-codecov left a comment

Swatinem commented Nov 7, 2024

adrian-codecov Nov 27, 2024

Swatinem Nov 28, 2024

Swatinem commented Nov 28, 2024

adrian-codecov left a comment

github-actions bot commented Nov 29, 2024

Store v2 reports in GCS instead of Redis #960

Store v2 reports in GCS instead of Redis #960

Conversation

Swatinem commented Nov 5, 2024 • edited Loading

codecov bot commented Nov 5, 2024 • edited Loading

Codecov Report

codecov-notifications bot commented Nov 5, 2024 • edited Loading

Codecov Report

codecov-qa bot commented Nov 5, 2024 • edited Loading

❌ 2 Tests Failed:

github-actions bot commented Nov 6, 2024

codecov-public-qa bot commented Nov 6, 2024 • edited Loading

❌ Failed Test Results:

pytest

matt-codecov left a comment

Choose a reason for hiding this comment

Swatinem commented Nov 7, 2024

adrian-codecov Nov 27, 2024

Choose a reason for hiding this comment

Swatinem Nov 28, 2024

Choose a reason for hiding this comment

Swatinem commented Nov 28, 2024

adrian-codecov left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 29, 2024

Swatinem commented Nov 5, 2024 •

edited

Loading

codecov bot commented Nov 5, 2024 •

edited

Loading

codecov-notifications bot commented Nov 5, 2024 •

edited

Loading

codecov-qa bot commented Nov 5, 2024 •

edited

Loading

codecov-public-qa bot commented Nov 6, 2024 •

edited

Loading