Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add history, fix percent_diff rounding and flake aggregate time #868

Merged
merged 3 commits into from
Oct 8, 2024

Conversation

suejung-sentry
Copy link
Contributor

@suejung-sentry suejung-sentry commented Oct 8, 2024

Ported changes from #857 with merge conflicts resolved

@codecov-staging
Copy link

codecov-staging bot commented Oct 8, 2024

Codecov Report

Attention: Patch coverage is 82.75862% with 10 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
graphql_api/types/repository/repository.py 73.07% 7 Missing ⚠️
graphql_api/types/test_analytics/test_analytics.py 90.32% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@codecov-qa
Copy link

codecov-qa bot commented Oct 8, 2024

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
2549 1 2548 6
View the top 1 failed tests by shortest run time
graphql_api.tests.test_test_analytics.TestAnalyticsTestCase test_flake_aggregates_7_days
Stack Traces | 0.334s run time
self = <graphql_api.tests.test_test_analytics.TestAnalyticsTestCase testMethod=test_flake_aggregates_7_days>

    def test_flake_aggregates_7_days(self) -> None:
        repo = RepositoryFactory(
            author=self.owner, active=True, private=True, branch="main"
        )
    
        test = TestFactory(repository=repo)
    
        _ = FlakeFactory(
            repository=repo,
            test=test,
            start_date=datetime.datetime.now() - datetime.timedelta(days=1),
            end_date=None,
        )
        _ = FlakeFactory(
            repository=repo,
            test=test,
            start_date=datetime.datetime.now() - datetime.timedelta(days=11),
            end_date=datetime.datetime.now() - datetime.timedelta(days=8),
        )
        _ = FlakeFactory(
            repository=repo,
            test=test,
            start_date=datetime.datetime.now() - datetime.timedelta(days=30),
            end_date=datetime.datetime.now() - datetime.timedelta(days=10),
        )
    
        for i in range(0, 7):
            _ = DailyTestRollupFactory(
                test=test,
                repoid=repo.repoid,
                branch="main",
                flaky_fail_count=1 if i % 7 == 0 else 0,
                fail_count=1 if i % 7 == 0 else 0,
                skip_count=0,
                pass_count=1,
                avg_duration_seconds=float(i),
                last_duration_seconds=float(i),
                date=datetime.date.today() - datetime.timedelta(days=i),
            )
        for i in range(7, 14):
            _ = DailyTestRollupFactory(
                test=test,
                repoid=repo.repoid,
                branch="main",
                flaky_fail_count=1 if i % 3 == 0 else 0,
                fail_count=1 if i % 3 == 0 else 0,
                skip_count=0,
                pass_count=1,
                avg_duration_seconds=float(i),
                last_duration_seconds=float(i),
                date=datetime.date.today() - datetime.timedelta(days=i),
            )
    
>       res = self.fetch_repository(
            repo.name,
            """flakeAggregates(history: INTERVAL_7_DAY) { flakeCount, flakeRate, flakeCountPercentChange, flakeRatePercentChange }""",
        )
E       AttributeError: 'TestAnalyticsTestCase' object has no attribute 'fetch_repository'

graphql_api/tests/test_test_analytics.py:882: AttributeError

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

Copy link

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 2555 tests with 1 failed, 2548 passed and 6 skipped.

View the full list of failed tests

pytest

  • Class name: graphql_api.tests.test_test_analytics.TestAnalyticsTestCase
    Test name: test_flake_aggregates_7_days

    self = <graphql_api.tests.test_test_analytics.TestAnalyticsTestCase testMethod=test_flake_aggregates_7_days>

    def test_flake_aggregates_7_days(self) -> None:
    repo = RepositoryFactory(
    author=self.owner, active=True, private=True, branch="main"
    )

    test = TestFactory(repository=repo)

    _ = FlakeFactory(
    repository=repo,
    test=test,
    start_date=datetime.datetime.now() - datetime.timedelta(days=1),
    end_date=None,
    )
    _ = FlakeFactory(
    repository=repo,
    test=test,
    start_date=datetime.datetime.now() - datetime.timedelta(days=11),
    end_date=datetime.datetime.now() - datetime.timedelta(days=8),
    )
    _ = FlakeFactory(
    repository=repo,
    test=test,
    start_date=datetime.datetime.now() - datetime.timedelta(days=30),
    end_date=datetime.datetime.now() - datetime.timedelta(days=10),
    )

    for i in range(0, 7):
    _ = DailyTestRollupFactory(
    test=test,
    repoid=repo.repoid,
    branch="main",
    flaky_fail_count=1 if i % 7 == 0 else 0,
    fail_count=1 if i % 7 == 0 else 0,
    skip_count=0,
    pass_count=1,
    avg_duration_seconds=float(i),
    last_duration_seconds=float(i),
    date=datetime.date.today() - datetime.timedelta(days=i),
    )
    for i in range(7, 14):
    _ = DailyTestRollupFactory(
    test=test,
    repoid=repo.repoid,
    branch="main",
    flaky_fail_count=1 if i % 3 == 0 else 0,
    fail_count=1 if i % 3 == 0 else 0,
    skip_count=0,
    pass_count=1,
    avg_duration_seconds=float(i),
    last_duration_seconds=float(i),
    date=datetime.date.today() - datetime.timedelta(days=i),
    )

    > res = self.fetch_repository(
    repo.name,
    """flakeAggregates(history: INTERVAL_7_DAY) { flakeCount, flakeRate, flakeCountPercentChange, flakeRatePercentChange }""",
    )
    E AttributeError: 'TestAnalyticsTestCase' object has no attribute 'fetch_repository'

    graphql_api/tests/test_test_analytics.py:882: AttributeError

Copy link

codecov bot commented Oct 8, 2024

Codecov Report

Attention: Patch coverage is 82.75862% with 10 lines in your changes missing coverage. Please review.

Project coverage is 96.29%. Comparing base (e7bf4d0) to head (0252042).
Report is 1 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
graphql_api/types/repository/repository.py 73.07% 7 Missing ⚠️
graphql_api/types/test_analytics/test_analytics.py 90.32% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #868      +/-   ##
==========================================
- Coverage   96.34%   96.29%   -0.05%     
==========================================
  Files         823      823              
  Lines       19161    19187      +26     
==========================================
+ Hits        18461    18477      +16     
- Misses        700      710      +10     
Flag Coverage Δ
unit 92.66% <82.75%> (-0.05%) ⬇️
unit-latest-uploader 92.66% <82.75%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@suejung-sentry suejung-sentry force-pushed the joseph-sshin/test-results-history branch 3 times, most recently from 0e5a17e to c7bdcf1 Compare October 8, 2024 00:28
@suejung-sentry suejung-sentry force-pushed the joseph-sshin/test-results-history branch from a9821f8 to 780fc27 Compare October 8, 2024 00:59
@joseph-sentry
Copy link
Contributor

Gonna address @ajay-sentry's feedback on this PR:

start date < until date but end date > since date?

Could you remind me again what start and end mean in this context?
I'd figure we'd want start > until and end < since but I could just be messing with my head

this one is a bit confusing but the idea is that we want flakes that were active at some point during the time period starting at since and ending at until so, if it started before until and it ended after since. This is also assuming that end > start which is an obvious assumption to make.

  1. if it started after until then it occured after the time period
  2. if it ended before start then it occured before the time period
  3. otherwise it occured during the time period

- remove unnecessary convert_kwargs_to_snake_case decorator call
@joseph-sentry joseph-sentry marked this pull request as ready for review October 8, 2024 15:59
@joseph-sentry joseph-sentry requested a review from a team as a code owner October 8, 2024 15:59
return await sync_to_async(generate_test_results_aggregates)(
repoid=repository.repoid
repoid=repository.repoid, history=history
)


@test_analytics_bindable.field("flakeAggregates")
@convert_kwargs_to_snake_case
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dont need these @convert_kwargs_to_snake_case

Copy link
Contributor

@ajay-sentry ajay-sentry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@joseph-sentry joseph-sentry added this pull request to the merge queue Oct 8, 2024
Merged via the queue into main with commit f4a178c Oct 8, 2024
14 of 19 checks passed
@joseph-sentry joseph-sentry deleted the joseph-sshin/test-results-history branch October 8, 2024 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants