Add an option to opt-out of “fixes”?

[gohanlon]

Testing of fix-its is a great part of swift-macro-testing. Compared to swift-syntax's testing tools, testing of diagnostics and fix-its is perhaps my most valued feature.

But, for many of my tests the “fixes” are repetitive and don’t strengthen the test. For example, the following test (with hypothetical testFixIts option) shows that when code contains both “access leak” and “duplicate label” mistakes, only the “access leak” diagnostics are emitted. The application of the fix-its is redundant with several other tests, and not interesting for this test.

func testAccessLeakAndCustomLabelConflictsWithAnotherLabel() {
  assertMacro(testFixIts: false) {
    """
    @MemberwiseInit(.public)
    struct S {
      @Init(label: "foo") let x: T
      @Init(label: "foo") let y: T
    }
    """
  } diagnostics: {
    """
    @MemberwiseInit(.public)
    struct S {
      @Init(label: "foo") let x: T
      ┬───────────────────────────
      ╰─ 🛑 @MemberwiseInit(.public) would leak access to 'internal' property
         ✏️ Add '@Init(.public)'
         ✏️ Add 'public' access level
         ✏️ Add '@Init(.ignore)' and an initializer
      @Init(label: "foo") let y: T
      ┬───────────────────────────
      ╰─ 🛑 @MemberwiseInit(.public) would leak access to 'internal' property
         ✏️ Add '@Init(.public)'
         ✏️ Add 'public' access level
         ✏️ Add '@Init(.ignore)' and an initializer
    }
    """
  }
}

Thoughts?

[stephencelis]

@gohanlon Something we've wanted to support in inline snapshots in general is more complex config.

So you could do something like the following to skip recording a particular snapshot inline:

func testAccessLeakAndCustomLabelConflictsWithAnotherLabel() {
  assertMacro(testFixIts: false) {
    """
    @MemberwiseInit(.public)
    struct S {
      @Init(label: "foo") let x: T
      @Init(label: "foo") let y: T
    }
    """
  } diagnostics: {
    """
    @MemberwiseInit(.public)
    struct S {
      @Init(label: "foo") let x: T
      ┬───────────────────────────
      ╰─ 🛑 @MemberwiseInit(.public) would leak access to 'internal' property
         ✏️ Add '@Init(.public)'
         ✏️ Add 'public' access level
         ✏️ Add '@Init(.ignore)' and an initializer
      @Init(label: "foo") let y: T
      ┬───────────────────────────
      ╰─ 🛑 @MemberwiseInit(.public) would leak access to 'internal' property
         ✏️ Add '@Init(.public)'
         ✏️ Add 'public' access level
         ✏️ Add '@Init(.ignore)' and an initializer
    }
    """
  }
} fixIts: {
  .skip
}

[gohanlon]

Very intersting! That would work for this case.

I think swift-macro-testing may still need specific configuration for "fixes", which would need to interact well with general inline snapshots configuration. For example, for tests where applying all the fix-its at once isn't valid (see #14), an enum configuration might make sense rather than a bool (e.g. testFixIts: .all, .none/nil, .separately).