Dependency resolution problem when compiling on OSX with GHC 8

[ashleyconnor]

So I know you guys don't officially support GHC 8 but I'm thinking if a minor change can be made to work for both GHC 7.10 and GHC 8 then it wouldn't hurt to do so.

Sierra, El Capitan, Yosemite build logs

==> cabal sandbox init
Config file path source is default config file.
Config file
/private/tmp/elm-20161115-81749-yjzkoa/elm-compiler-0.18.0/.cabal/config not
found.
Writing default configuration to
/private/tmp/elm-20161115-81749-yjzkoa/elm-compiler-0.18.0/.cabal/config
Writing a default package environment file to
/private/tmp/elm-20161115-81749-yjzkoa/elm-compiler-0.18.0/cabal.sandbox.config
Creating a new sandbox at
/private/tmp/elm-20161115-81749-yjzkoa/elm-compiler-0.18.0/.cabal-sandbox
==> cabal update
Downloading the latest package list from hackage.haskell.org
==> cabal sandbox add-source elm-compiler elm-package elm-make elm-repl elm-reactor
==> cabal install --jobs=4 --max-backjumps=100000 --only-dependencies elm-compiler elm-package elm-make elm-repl elm-reactor
clang: warning: -Wl,-headerpad_max_install_names: 'linker' input unused
clang: warning: argument unused during compilation: '-L/usr/local/lib'
clang: warning: argument unused during compilation: '-L/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries'
Resolving dependencies...
cabal: Could not resolve dependencies:
trying: transformers-0.5.2.0/installed-0.5... (dependency of union-find-0.2)
trying: elm-package-0.18 (user goal)
next goal: optparse-applicative (dependency of elm-package-0.18)
rejecting: optparse-applicative-0.13.0.0, optparse-applicative-0.12.1.0,
optparse-applicative-0.12.0.0 (conflict: elm-package =>
optparse-applicative>=0.11 && <0.12)
rejecting: optparse-applicative-0.11.0.2, optparse-applicative-0.11.0.1,
optparse-applicative-0.11.0 (conflict: transformers==0.5.2.0/installed-0.5...,
optparse-applicative => transformers>=0.2 && <0.5)
rejecting: optparse-applicative-0.10.0, optparse-applicative-0.9.1.1,
optparse-applicative-0.9.1, optparse-applicative-0.9.0,
optparse-applicative-0.8.1, optparse-applicative-0.8.0.1,
optparse-applicative-0.8.0, optparse-applicative-0.7.0.2,
optparse-applicative-0.7.0.1, optparse-applicative-0.7.0,
optparse-applicative-0.6.0, optparse-applicative-0.5.2.1,
optparse-applicative-0.5.2, optparse-applicative-0.5.1,
optparse-applicative-0.5.0, optparse-applicative-0.4.3,
optparse-applicative-0.4.2, optparse-applicative-0.4.1,
optparse-applicative-0.4.0, optparse-applicative-0.3.2,
optparse-applicative-0.3.1, optparse-applicative-0.3.0,
optparse-applicative-0.2.0, optparse-applicative-0.1.1,
optparse-applicative-0.1.0, optparse-applicative-0.0.1 (conflict: elm-package
=> optparse-applicative>=0.11 && <0.12)
Dependency tree exhaustively searched.

[process-bot]

Thanks for the issue! Make sure it satisfies this checklist. My human colleagues will appreciate it!

Here is what to expect next, and if anyone wants to comment, keep these things in mind.

[evancz]

What is the goal of this issue? What are we actually tracking here and is there any concrete work that it is valuable to do?

[ashleyconnor]

What is the goal of this issue?

Honestly I just wanted to get elm 0.18.0 running with homebrew so I could test out the new debugging features. I'm not one to install software via .pkg files.

The issue was opened because the homebrew project likes to report problems upstream in the hope that the package maintainers will incorporate fixes so changes don't need to be applied in homebrew recipes themselves.

What are we actually tracking here and is there any concrete work that it is valuable to do?

As far as value to yourself and the project, staying on top of issues compiling with GHC 8 would make it easier to upgrade when you decide that GHC 8 should be a min requirement for the elm project.

[evancz]

Elm 0.18 is built with GHC 7.10. It was already released, so that code is done. It has tags at exact commits that will not change. These constraints on GHC are also checked into our build script, so based on the information you have provided, I do not know how folks are building things wrong in the first place.

So I don't meant to sound combative here, but I also do not understand why trying to build software with compiler versions that are unsupported is an upstream issue. We may move to later versions of GHC at some point, but I have heard that compile times have gone up considerably in certain cases right now.

[evancz]

Maybe here's another way to ask the root question. You say:

I'm thinking if a minor change can be made to work for both GHC 7.10 and GHC 8 then it wouldn't hurt to do so.

Can you be more specific than "wouldn't hurt"? I don't know all the things you know, so I feel that there's some implicit information that you are assuming was conveyed already. If that is as specific as this can be, we should close the issue because I understand and keeping this issue open would not help organize or expedite the process.

[ashleyconnor]

Can you be more specific than "wouldn't hurt"?

I was literally thinking of fixing to a version of optparse-applicative that would allow build with GHC 7.10 and GHC 8, that's all. By "wouldn't hurt" I meant not requiring major changes, just fixing a dependency version. I'm not aware if there is even a version of optparse-applicative that would work with both and in that case I wouldn't make any changes at all to support GHC 8.

I'm going to close this issue. I appreciate you taking the time to look at this and explaining your reasoning.

[evancz]

No problem, and sorry if I sound cranky!

[ilovezfs]

@evancz Homebrew maintainer here. Since macOS 10.12 Sierra requires GHC 8, and because Homebrew only supports GHC 8, I'm thinking we will need to boneyard elm at this point until there's actually upstream support for GHC 8. We've been building elm with GHC 8 for 6 months now, with approximately 600 installs per month without a problem, so hopefully you will reconsider your stance.

[evancz]

I have no stance, only trying to figure out priorities. I need a better reason than "it wouldn't hurt" to change my development environment in a serious way, which you have provided.

"Sierra requires GHC 8" does not seem right to me. I'm using GHC 7.10 on macOS 10.12 Sierra. I built the Elm 0.18 binaries with it on this computer as well.

But accepting that you need it to be GHC 8 for whatever reason, how would you suggest resolving the fact that Elm 0.18 is already tagged and released? We can make changes in subsequent releases, but I do not know enough about your project to know what you actually need right now. What specifically needs to happen for Elm 0.18 to be available though homebrew?

[ilovezfs]

I'm using GHC 7.10 on macOS 10.12 Sierra, so I don't know exactly what you mean by "Sierra requires GHC 8".

I'm referring to https://ghc.haskell.org/trac/ghc/ticket/12479 which has been resolved by https://phabricator.haskell.org/D2611 in combination with haskell/cabal#3979 (except for cabal new-build, which is still broken).

The Sierra bottle in Homebrew is currently 8.0.1.20161114 and has both the GHC and cabal sides of that fix.

how would you suggest resolving the fact that Elm 0.18 is already tagged and released?

Any github commit can be converted to a patch file by appending .patch so if there are commits in your repo to the elm cabal files, we can apply them as patches to the release tarballs assuming the patches apply cleanly.

If the patches do not apply cleanly, we can manually commit an equivalent patch that does apply cleanly to our formula-patches repository, and use that.

Alternatively, the relevant fixes can be made using string replacements when that's more convenient, but the outcome is no different from a patch except in terms of the mechanism.

The final option is what we've been doing for the last six months, which is to write a cabal.config file to the root of the cabal sandbox we build in, with the contents

allow-newer: base,time,transformers,HTTP

@ashleyconnor found that for his recent elm 0.18 byuld, aeson-pretty needed to be added to that list as well.

The bottom line is that if you have elm building with GHC 8 successfully in your GitHub repos, there are many ways for us to backport that to 0.18.

[evancz]

Thanks for explaining these things to me! If I understand correctly, is one solution is to change that cabal.config file to the following?

allow-newer: base,time,transformers,HTTP,aeson-pretty

This seems like a safer way to go than the manual patching route. How do you evaluate this tradeoff from your perspective?

[ilovezfs]

Thanks for explaining these things to me!

You're welcome.

If I understand correctly, is one solution is to change that cabal.config file to the following?

Yep, and that builds.

So Solver's choices with that file in place should help to determine which constraints in the current elm cabal files are causing the cabal.config overrides to be necessary.

[evancz]

I made an edit that I think got lost. Using cabal.config seems safer than doing the patching route, and as I understand, it already works. I'm curious if you agree that this is actually a relatively nice solution compared to the patching idea.

From there, are there any concrete actions that I must take to get Elm to work with homebrew? If so, what are they exactly? If not, it sounds like we are all set.

[ilovezfs]

@evancz no the cabal.config approach is unacceptable except as a very short term stopgap measure, since it globally overrides the constraints in an unpredictable manner. This needs to be fixed with GHC 8 compatible settings in the elm cabal files.

[evancz]

You do not build the project with cabal sandboxes? I use that approach so no build can mess with anything else on my computer, and without it, global projects can conflict whether you use cabal.config or not.

Is it not possible to use cabal sandboxes? Or is it undesirable for some reason?

[ilovezfs]

We use cabal sandboxes for everything, but a global cabal.config for the sandbox elm builds in amounts to downstream patching, which is not acceptable.

[evancz]

Here is a cabal.config from elm-compiler that has exact constraints for everything. So it's not unpredictable at all. Once you get it to build, you are good to go for everyone ever. They will get exactly the same constraints.

If I was building any Haskell project again and again as an install path in a sandbox, I think I'd want to pin the constraints like this to avoid getting broken by a patch release (which can happen in Haskell). So this seems like work you'd already want to do for reliability.

So like I said before:

From there, are there any concrete actions that I must take to get Elm to work with homebrew? If so, what are they exactly? If not, it sounds like we are all set.

[ilovezfs]

Your elm-compiler.cabal, elm-reactor.cabal, etc. need to be compatible with GHC 8 or we've actually made no progress in addressing this upstream.

[evancz]

Your elm-compiler.cabal, elm-reactor.cabal, etc. need to be compatible with GHC 8

But from before:

If I understand correctly, is one solution is to change that cabal.config file to the following?

Yep, and that builds.

And it seems we agree that having a cabal.config is actually necessary for creating reliable builds independent of changes in the hackage ecosystem.

So I don't understand what we are discussing here. If there is a problem, let's work on it instead of continuing this thread. Create a new issue that describes the problem clearly, and the stakes of the problem, and that work can be prioritized along with all the other things that people want.

[ilovezfs]

But from before:

The fact that a cabal.config file inserted downstream can force it to build does not mean elm builds without downstream interventions, which is the requirement.

And it seems we agree that having a cabal.config is actually necessary for creating reliable builds independent of changes in the hackage ecosystem.

This is not something we agree on. This is the extent of the cabal.config usage we have currently, all of which are stop-gap measures pending proper upstream resolution:

Josephs-MacBook-Pro:Formula joe$ ag cabal.config
cless.rb
28:    (buildpath/"cabal.config").write("allow-newer: base,transformers\n")

elm.rb
59:    (buildpath/"cabal.config").write("allow-newer: base,time,transformers,HTTP\n")

pandoc-crossref.rb
24:    (buildpath/"cabal.config").write("allow-newer: pandoc,pandoc-types\n")

In the case of elm, the problem was brought to your attention six months ago, and continuing to work around the incompatibility with downstream hacks is not acceptable.

If there is a problem, let's work on it instead of continuing this thread.

This is already the second of two issues bringing the GHC 8 incompatibility of your build system to your attention.

[ilovezfs]

@evancz Also, here's what we get with the cabal.config from your gist:

==> cabal install --jobs=8 --max-backjumps=100000 --only-dependencies elm-compiler elm-package elm-make elm-repl elm-reactor
clang: warning: -Wl,-headerpad_max_install_names: 'linker' input unused
clang: warning: argument unused during compilation: '-L/usr/local/lib'
clang: warning: argument unused during compilation: '-L/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries'
Resolving dependencies...
cabal: Could not resolve dependencies:
trying: elm-compiler-0.18 (user goal)
next goal: base (dependency of elm-compiler-0.18)
rejecting: base-4.9.0.0/installed-4.9..., base-4.9.0.0, base-4.8.2.0
(constraint from user config
/private/tmp/elm-20161116-51455-7g0rzp/elm-compiler-0.18.0/cabal.config
requires ==4.8.1.0)
rejecting: base-4.8.1.0 (constraint from non-upgradeable package requires
installed instance)
rejecting: base-4.8.0.0, base-4.7.0.2, base-4.7.0.1, base-4.7.0.0,
base-4.6.0.1, base-4.6.0.0, base-4.5.1.0, base-4.5.0.0, base-4.4.1.0,
base-4.4.0.0, base-4.3.1.0, base-4.3.0.0, base-4.2.0.2, base-4.2.0.1,
base-4.2.0.0, base-4.1.0.0, base-4.0.0.0, base-3.0.3.2, base-3.0.3.1
(constraint from user config
/private/tmp/elm-20161116-51455-7g0rzp/elm-compiler-0.18.0/cabal.config
requires ==4.8.1.0)
Dependency tree exhaustively searched.

Note: when using a sandbox, all packages are required to have consistent
dependencies. Try reinstalling/unregistering the offending packages or
recreating the sandbox.
/usr/local/Homebrew/Library/Homebrew/debrew.rb:11:in `raise'
BuildError: Failed executing: cabal install --jobs=8 --max-backjumps=100000 --only-dependencies elm-compiler elm-package elm-make elm-repl elm-reactor
1. raise
2. ignore
3. backtrace
4. irb
5. shell

[ilovezfs]

@evancz OK, so putting aside that cabal.config (which causes conflicts versus the GHC 8 preinstalled versions for base, integer-gmp, ghc-prim, transformers, deepseq, array, process, unix, time, template-haskell, filepath, directory, containers, bytestring, and pretty) it looks like the constraints actually causing the Solver issues in your current cabal files are

    inreplace "elm-make/elm-make.cabal" do |s|
      s.gsub! "optparse-applicative >=0.11 && <0.12,", "optparse-applicative >=0.11 && <0.14," # 0.13.0.0 is current
    end

    inreplace "elm-package/elm-package.cabal" do |s|
      s.gsub! "optparse-applicative >= 0.11 && < 0.12,", "optparse-applicative >= 0.11 && < 0.14," # 0.13.0.0 is current
      s.gsub! "HTTP >= 4000.2.5 && < 4000.3,", "HTTP >= 4000.2.5 && < 4000.4," # 4000.3.3 is current
    end

If I just make those replacements, and don't use the cabal.config file you provided, then it builds.

[ilovezfs]

@evancz Here is a complete log https://gist.github.com/ilovezfs/99296a4a32419519845be9ba44958f24 and two PRs:

elm-lang/elm-make#130
elm-lang/elm-package#252

[evancz]

With these two changes, things build with GHC 8? And you are happy about how everything would be?

(Also, that cabal.config is the one from elm-compiler I built with GHC 7.10. The point was that you can pin things in that file so it doesn't mess with global things. Sorry it took you down a weird path.)

[ilovezfs]

With these two changes, things build with GHC 8?

Yep, both 8.0.1 and 8.0.1.20161114 (which is very close to what 8.0.2 will be).

And you are happy about how everything would be?

Yup, I'm happy if you're satisfied that the newer versions don't actually break anything. It chose optparse-applicative-0.13.0.0 and HTTP-4000.3.3. It would be even better if you could tag an 0.18.1 or 0.18.0.1 but the string replacements in Homebrew/homebrew-core@2a5650a are acceptable if you've merged (or intend to merge) those changes.

It would be great if you could enable CI for GHC 8 to prevent future regressions. Also you should personally use our Sierra bottle! And I'd like a pony.

[evancz]

Yeah, those changes should be okay! I'll be able to solve to something else if needed.

I forget why we disabled GHC 8 in CI, but I'm happy to have it turned back on. I'm not the primary expert in that stuff though, so maybe someone can help figure that out.

[ilovezfs]

Yeah, those changes should be okay! I'll be able to solve to something else if needed.

Excellent.

I forget why we disabled GHC 8 in CI, but I'm happy to have it turned back on. I'm not the primary expert in that stuff though, so maybe someone can help figure that out.

That would be great. Here's a rather fancy travis file for multiple GHC versions in case that helps:

https://github.com/idris-lang/Idris-dev/blob/master/.travis.yml