Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test: Roslyn.VisualStudio.IntegrationTests.CSharp.CSharpCodeActions.FastDoubleInvoke #57551

Closed
runfoapp bot opened this issue Nov 3, 2021 · 12 comments
Closed

Flaky test: Roslyn.VisualStudio.IntegrationTests.CSharp.CSharpCodeActions.FastDoubleInvoke #57551

runfoapp bot opened this issue Nov 3, 2021 · 12 comments
Assignees
@CyrusNajmabadi
Labels
Area-IDE Flaky Resolution-Fixed The bug has been fixed and/or the requested behavior has been implemented

Comments

@runfoapp
Copy link

runfoapp bot commented Nov 3, 2021 •
edited
Loading

Runfo Tracking Issue: Flaky test: Roslyn.VisualStudio.IntegrationTests.CSharp.CSharpCodeActions.FastDoubleInvoke

Build Definition Kind Run Name
1605523 roslyn-integration-CI PR 59168 1-Integration Debug OOP64_true OOPCoreClr_false
1605523 roslyn-integration-CI PR 59168 1-Integration Debug OOP64_true OOPCoreClr_false
1603866 roslyn-integration-CI PR 59443 1-Integration Debug OOP64_true OOPCoreClr_false
1603848 roslyn-integration-CI Rolling 1-Integration Debug OOP64_true OOPCoreClr_false
1603780 roslyn-integration-CI PR 59440 1-Integration Debug OOP64_true OOPCoreClr_false
1603695 roslyn-integration-CI Rolling 1-Integration Release OOP64_true OOPCoreClr_false
1603168 roslyn-integration-CI Rolling 1-Integration Release OOP64_true OOPCoreClr_false
1603117 roslyn-integration-CI PR 59426 1-Integration Debug OOP64_true OOPCoreClr_false
1602725 roslyn-integration-CI Rolling 1-Integration Release OOP64_true OOPCoreClr_false
1601915 roslyn-integration-CI PR 37994 1-Integration Release OOP64_false OOPCoreClr_false
1601421 roslyn-integration-CI Rolling 1-Integration Release OOP64_true OOPCoreClr_false
1600741 roslyn-integration-CI PR 59385 1-Integration Debug OOP64_true OOPCoreClr_false
1599283 roslyn-integration-CI Rolling 1-Integration Debug OOP64_true OOPCoreClr_false
1595064 roslyn-integration-CI PR 59320 1-Integration Debug OOP64_true OOPCoreClr_false
1593012 roslyn-integration-CI Rolling 1-Integration Release OOP64_true OOPCoreClr_false
1588730 roslyn-integration-CI Rolling 1-Integration Debug OOP64_true OOPCoreClr_false
1588233 roslyn-integration-CI PR 59100 1-Integration Debug OOP64_true OOPCoreClr_false
1588106 roslyn-integration-CI Rolling 1-Integration Debug OOP64_true OOPCoreClr_false
1581598 roslyn-integration-CI PR 57172 1-Integration Release OOP64_false OOPCoreClr_false
1581598 roslyn-integration-CI PR 57172 1-Integration Release OOP64_false OOPCoreClr_false
1581598 roslyn-integration-CI PR 57172 1-Integration Debug OOP64_true OOPCoreClr_false
1581598 roslyn-integration-CI PR 57172 1-Integration Debug OOP64_true OOPCoreClr_false
1580981 roslyn-integration-CI PR 59096 1-Integration Debug OOP64_true OOPCoreClr_false
1577263 roslyn-integration-CI PR 59073 1-Integration Release OOP64_false OOPCoreClr_false
1577034 roslyn-integration-CI PR 59073 1-Integration Release OOP64_false OOPCoreClr_false
1577034 roslyn-integration-CI PR 59073 1-Integration Debug OOP64_true OOPCoreClr_false
1576316 roslyn-integration-CI PR 59073 1-Integration Release OOP64_false OOPCoreClr_false
1575020 roslyn-integration-CI Rolling 1-Integration Release OOP64_true OOPCoreClr_false
1574333 roslyn-integration-CI PR 59086 1-Integration Debug OOP64_true OOPCoreClr_false
1574054 roslyn-integration-CI PR 59073 1-Integration Debug OOP64_true OOPCoreClr_false
1569244 roslyn-integration-CI PR 59036 1-Integration Debug OOP64_true OOPCoreClr_false
1566265 roslyn-integration-CI Rolling 1-Integration Debug OOP64_true OOPCoreClr_false
1565971 roslyn-integration-CI PR 58975 1-Integration Debug OOP64_true OOPCoreClr_false
1563464 roslyn-integration-CI Rolling 1-Integration Release OOP64_false OOPCoreClr_false
1563247 roslyn-integration-CI PR 58968 1-Integration Debug OOP64_true OOPCoreClr_false
1562105 roslyn-integration-CI PR 58960 1-Integration Debug OOP64_true OOPCoreClr_false
1559254 roslyn-integration-CI PR 58927 1-Integration Debug OOP64_true OOPCoreClr_false
1554580 roslyn-integration-CI PR 50569 1-Integration Debug OOP64_true OOPCoreClr_false
1554285 roslyn-integration-CI Rolling 1-Integration Release OOP64_true OOPCoreClr_false
1554266 roslyn-integration-CI Rolling 1-Integration Release OOP64_true OOPCoreClr_false
1552232 roslyn-integration-CI PR 58810 1-Integration Debug OOP64_true OOPCoreClr_false
1551977 roslyn-integration-CI PR 58852 1-Integration Debug OOP64_true OOPCoreClr_false

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
4 14 37
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged Issues and PRs which have not yet been triaged by a lead label Nov 3, 2021
@jasonmalinowski jasonmalinowski changed the title Flaky test: FastDoubleInvoke Flaky test: Roslyn.VisualStudio.IntegrationTests.CSharp.CSharpCodeActions.FastDoubleInvoke Nov 3, 2021
@jasonmalinowski jasonmalinowski added Flaky Area-IDE and removed untriaged Issues and PRs which have not yet been triaged by a lead labels Nov 3, 2021
@jasonmalinowski
Copy link
Member

Another possible dupe of the bugs @CyrusNajmabadi is looking at.

@CyrusNajmabadi
Copy link
Member

What version of vs is ci running?

@jasonmalinowski
Copy link
Member

@CyrusNajmabadi 17.0 something or another; is there a fix in 17.1?

@JoeRobich
Copy link
Member

Here is the VS2022 upgrade schedule for our CI images https://github.com/dotnet/core-eng/wiki/VS2022-Upgrade-Schedule

@CyrusNajmabadi
Copy link
Member

@jasonmalinowski yes, there are many fixes to async lightbulbs in 17.1 (both on our side and on the editor). Can we move to a 17.1 build?

@jasonmalinowski
Copy link
Member

@JoeRobich would best know the schedule; usually we can get an image up about a week after it goes public.

@JoeRobich
Copy link
Member

JoeRobich commented Nov 5, 2021 •
edited
Loading

See link to schedule above.

Typical order of events:

  • VS preview released on a Tuesday
  • dnceng updates their image for new release
  • New image is generated (mix of automatic and manual configuration) and in staging pool by EOW
  • Image is validated early following week
  • Image is rolled out on the Wednesday the week after release for our use

This order of events changes when/if they introduce the Scouting queue. An additional week will be added for the image to live in the scouting queue for validation by teams before fully rolling out.

@CyrusNajmabadi
Copy link
Member

What mechanism do we have though if we have to absorb a fix from another team to deal with CI flakeyness? Say, for example, that platform's fixes here don't fully address the issue. We don't want to wait for a new preview to get our CI clean, and we def don't want to disable thse tests.

@CyrusNajmabadi
Copy link
Member

@sharwell @JoeRobich are we still seeing htis?

@sharwell
Copy link
Member

sharwell commented Dec 20, 2021 •
edited
Loading

Yes, this situation can still occur. The test itself is conceptually flawed, such that a correct implementation will necessarily be flaky.

I am aware of two assumptions the test makes which are not required to hold at run time:

  1. If a user presses Ctrl+. on a line which is not currently reporting a Simplify Type Name diagnostic, the code fix infrastructure is not required to run the Simplify Type Name analyzer to determine if there are any missing diagnostics which could have been visible. This is required to achieve the performance improvements described by DiagnosticAnalyzer implementations need a syntax filter to exclude their overhead from unrelated light bulbs #55228.

  2. The light bulb infrastructure does not notify the editor of a change to light bulb options when the editor text changes; rather, it waits for information from a subsequent analyzer pass. The editor is therefore not required to re-query suggested actions between the time an edit is made and the time the next diagnostic pass completes. The following reports the event for DiagnosticsUpdated, but notice how there is no event for WorkspaceChanged→DocumentChanged:

    roslyn/src/EditorFeatures/Core.Wpf/Suggestions/SuggestedActionsSource.cs

    Lines 538 to 547 in 1c3ac57

    private void OnDiagnosticsUpdated(object sender, DiagnosticsUpdatedArgs e)
    {
    // document removed case. no reason to raise event
    if (e.Solution == null)
    {
    return;
    }
    OnSuggestedActionsChanged(e.Workspace, e.DocumentId, e.Solution.WorkspaceVersion);
    }

@JoeRobich
Copy link
Member

Test is removed in #59457

@sharwell sharwell added the Resolution-Fixed The bug has been fixed and/or the requested behavior has been implemented label Feb 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CyrusNajmabadi CyrusNajmabadi

Labels
Area-IDE Flaky Resolution-Fixed The bug has been fixed and/or the requested behavior has been implemented
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jasonmalinowski @JoeRobich @sharwell @CyrusNajmabadi