Crash in CodeFixTest`1.FixEachAnalyzerDiagnosticAsync

[mavasani]

Add the below unit test to https://github.com/dotnet/roslyn-analyzers/blob/master/src/PublicApiAnalyzers/UnitTests/NullableEnablePublicApiAnalyzerTests.cs

        [Fact]
        public async Task NullableEnableShippedAPI_FixAll()
        {
            var source = @"
#nullable enable
public class C1
{
    public string? Field;
}
";
            var shippedText1 = @"";
            var unshippedText = @"C1
C1.C1() -> void
C1.Field -> string";

            var shippedText2 = @"";

            var fixedShippedText = @"#nullable enable
";

            var test = new CSharpCodeFixTest<DeclarePublicApiAnalyzer, NullableEnablePublicApiFix, XUnitVerifier>();

            test.TestState.Sources.Add(source);
            test.TestState.AdditionalFiles.Add((DeclarePublicApiAnalyzer.ShippedFileName, shippedText1));
            test.TestState.AdditionalFiles.Add((DeclarePublicApiAnalyzer.UnshippedFileName, unshippedText));

            test.SolutionTransforms.Add(
                (solution, projectId) =>
                {
                    var project1 = solution.GetProject(projectId);
                    var project2 = solution.AddProject("Project2", "Project2", LanguageNames.CSharp)
                        .AddDocument("File.cs", source).Project
                        .AddAdditionalDocument(DeclarePublicApiAnalyzer.ShippedFileName, shippedText2).Project
                        .AddAdditionalDocument(DeclarePublicApiAnalyzer.UnshippedFileName, unshippedText).Project
                        .AddMetadataReferences(project1.MetadataReferences)
                        .WithCompilationOptions(project1.CompilationOptions);

                    return project2.Solution;
                });

            test.TestState.ExpectedDiagnostics.AddRange(new[]
            {
                // /0/Test0.cs(5,20): warning RS0037: PublicAPI.txt is missing '#nullable enable', so the nullability annotations of API isn't recorded. It is recommended to enable this tracking.
                new DiagnosticResult(DeclarePublicApiAnalyzer.ShouldAnnotateApiFilesRule).WithSpan(5, 20, 5, 25),
                // File.cs(5,20): warning RS0037: PublicAPI.txt is missing '#nullable enable', so the nullability annotations of API isn't recorded. It is recommended to enable this tracking.
                new DiagnosticResult(DeclarePublicApiAnalyzer.ShouldAnnotateApiFilesRule).WithSpan("File.cs", 5, 20, 5, 25),
            });

            test.FixedState.Sources.Add(source);
            test.FixedState.AdditionalFiles.Add((DeclarePublicApiAnalyzer.ShippedFileName, fixedShippedText));
            test.FixedState.AdditionalFiles.Add((DeclarePublicApiAnalyzer.UnshippedFileName, unshippedText));

            test.FixedState.ExpectedDiagnostics.AddRange(new[]
            {
                // /0/Test0.cs(5,20): warning RS0036: Symbol 'Field' is missing nullability annotations in the declared API.
                new DiagnosticResult(DeclarePublicApiAnalyzer.AnnotateApiRule).WithSpan(5, 20, 5, 25).WithArguments("Field"),
                // File.cs(5,20): warning RS0037: PublicAPI.txt is missing '#nullable enable', so the nullability annotations of API isn't recorded. It is recommended to enable this tracking.
                new DiagnosticResult(DeclarePublicApiAnalyzer.ShouldAnnotateApiFilesRule).WithSpan("File.cs", 5, 20, 5, 25),

            });

            await test.RunAsync();
        }

Got the below null reference exception in test execution:

 System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace: 
    CodeFixContext.ctor(Document document, TextSpan span, ImmutableArray`1 diagnostics, Action`2 registerCodeFix, Boolean verifyArguments, CancellationToken cancellationToken)
    CodeFixContext.ctor(Document document, Diagnostic diagnostic, Action`2 registerCodeFix, CancellationToken cancellationToken)
    CodeFixTest`1.FixEachAnalyzerDiagnosticAsync(ImmutableArray`1 analyzers, ImmutableArray`1 codeFixProviders, Nullable`1 codeFixIndex, String codeFixEquivalenceKey, Project project, Int32 numberOfIterations, IVerifier verifier, CancellationToken cancellationToken) line 495
    CodeFixTest`1.VerifyFixAsync(String language, ImmutableArray`1 analyzers, ImmutableArray`1 codeFixProviders, SolutionState oldState, SolutionState newState, Int32 numberOfIterations, Func`9 getFixedProject, IVerifier verifier, CancellationToken cancellationToken) line 404
    CodeFixTest`1.VerifyFixAsync(SolutionState testState, SolutionState fixedState, SolutionState batchFixedState, IVerifier verifier, CancellationToken cancellationToken) line 371
    CodeFixTest`1.RunAsync(CancellationToken cancellationToken) line 247
    NullableEnablePublicApiAnalyzerTests.NullableEnableShippedAPI_FixAll() line 263
    --- End of stack trace from previous location where exception was thrown ---

[mavasani]

@sharwell

[sharwell]

It looks like this occurs if you have a code fix provider registered for a diagnostic which gets reported in an additional file, as opposed to a source file. The scenario was missed in testing because it doesn't work in the UI either.

[sharwell]

The error occurs because GetDocument returns null on this line:

roslyn-sdk/src/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.CodeFix.Testing/CodeFixTest`1.cs

Line 495 in 93b092f

var context = new CodeFixContext(project.GetDocument(diagnostic.Location.SourceTree), diagnostic, (a, d) => actions.Add(a), cancellationToken);

The solution is updating fixableDiagnostics to only include diagnostics appearing in a document here:

roslyn-sdk/src/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.CodeFix.Testing/CodeFixTest`1.cs

Lines 472 to 473 in 93b092f

	var fixableDiagnostics = analyzerDiagnostics.Where(
	diagnostic => codeFixProviders.Any(provider => provider.FixableDiagnosticIds.Contains(diagnostic.Id))).ToImmutableArray();