Solver doesn't always add stanza choices to the conflict set.

[grayjay]

I wrote a test case, but I haven't tried to debug it yet:
grayjay@71091b6

-- | This test demonstrates a bug in stanza preferences. The solver should be
-- able to install 'A' by enabling 'flag' and disabling testing. When it tries
-- goals in the specified order and prefers testing, it encounters
-- 'unknown-pkg2'. 'unknown-pkg2' is only introduced by testing and 'flag', so
-- the conflict set should contain both of those variables. However, it only
-- contains 'flag'. The solver backjumps past the choice to disable testing and
-- fails to find the solution.
testStanzaPreference :: String -> TestTree
testStanzaPreference name =
  let pkg = exAv "A" 1    [exFlag "flag"
                              []
                              [ExAny "unknown-pkg1"]]
             `withTest`
            ExTest "test" [exFlag "flag"
                              [ExAny "unknown-pkg2"]
                              []]
      goals = [
          P None "A"
        , F None "A" "flag"
        , S None "A" TestStanzas
        ]
  in runTest $ goalOrder goals $
     preferences [ ExStanzaPref "A" [TestStanzas]] $
     mkTest [Right pkg] name ["A"] $
     solverSuccess [("A", 1)]

The conflict set at the line [__2] fail (backjumping, conflict set: unknown-pkg2, A-1.0.0:flag) should also contain pkg-1.0.0:test:

targets: A
constraints: 
preferences: 
  A [TestStanzas]
strategy: PreferLatestForSelected
reorder goals: False
count conflicts: True
independent goals: False
avoid reinstalls: False
shadow packages: False
strong flags: False
max backjumps: infinite
[__0] trying: A-1.0.0 (user goal)
[__1] trying: A-1.0.0:+flag
[__2] trying: A-1.0.0:*test
[__3] unknown package: unknown-pkg2 (dependency of A-1.0.0:+flag)
[__2] fail (backjumping, conflict set: unknown-pkg2, A-1.0.0:flag)
[__1] trying: A-1.0.0:-flag
[__2] trying: A-1.0.0:*test
[__3] unknown package: unknown-pkg1 (dependency of A-1.0.0:-flag)
[__2] fail (backjumping, conflict set: unknown-pkg1, A-1.0.0:flag)
[__0] fail (backjumping, conflict set: A, unknown-pkg1, unknown-pkg2, A-1.0.0:flag)
Unit Tests
  UnitTests.Distribution.Solver.Modular.Solver
    Stanza Preferences
      test stanza preference: FAIL (0.03s)
        Unexpected error:
        Could not resolve dependencies:
        trying: A-1.0.0:+flag
        unknown package: unknown-pkg2 (dependency of A-1.0.0:+flag)
        Dependency tree exhaustively searched.

[kosmikus]

Summarizing some discussion I had with @grayjay:

This is a real bug. test/bench used to never be determined by the solver. They used to be manually resolved. And then it does not matter, because no other decision could ever be made for a test/bench goal. Now, we treat test/bench essentially as the treat automatic flags, so everything changes.

[ezyang]

I'm milestoning this as triaged, as in, it's a real bug, but not a release blocker.

[grayjay]

I also think it would be better to fix #3923 first. Fixing this bug will make it easier for the solver to disable tests, which could cause #3923 to show up more often.