Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stack docker does not work with GHC HEAD (ghc-pkg dump format change) #1509

Closed
CRogers opened this issue Dec 13, 2015 · 11 comments
Closed

Stack docker does not work with GHC HEAD (ghc-pkg dump format change) #1509

CRogers opened this issue Dec 13, 2015 · 11 comments

Comments

@CRogers
Copy link
Contributor

CRogers commented Dec 13, 2015

I've made a docker image of the optimised release build of GHC HEAD (which I intend to trigger/push nightly builds of) but it sadly doesn't work with stack. It seems that ghc-pkg lists dependencies in a different format than stack is expecting.

Steps to reproduce:

Expected:

  • The project builds, using the HEAD version of ghc inside the docker container.

Actual:

The version of stack I'm running:

$ stack --version
Version 0.1.10.0 x86_64

I'm on OS X running with docker-machine and docker 1.9.1. The command fails with this output:

$ stack --skip-ghc-check -v build
Version 0.1.10.0 x86_64
2015-12-13 17:36:38.330166: [debug] Checking for project config at: /Users/callumr/Dropbox/Programming/Haskell/ghc-head/stack-test/stack.yaml @(stack_15maE5vLiHaHnQnC7u59lN:Stack.Config src/Stack/Config.hs:576:9)
2015-12-13 17:36:38.330435: [debug] Loading project config file stack.yaml @(stack_15maE5vLiHaHnQnC7u59lN:Stack.Config src/Stack/Config.hs:599:13)
2015-12-13 17:36:38.332695: [debug] Run process: docker --version @(stack_15maE5vLiHaHnQnC7u59lN:System.Process.Read src/System/Process/Read.hs:273:3)
2015-12-13 17:36:38.340489: [debug] Run process: docker inspect 2be8010a53ad @(stack_15maE5vLiHaHnQnC7u59lN:System.Process.Read src/System/Process/Read.hs:273:3)
2015-12-13 17:36:38.500771: [debug] Run process: docker create --net=host -e STACK_IN_CONTAINER=1 -e STACK_ROOT=/Users/callumr/.stack -e STACK_PLATFORM_VARIANT=dkb98455da5dd905b08a4ce6671795697f -e HOME=/Users/callumr/Dropbox/Programming/Haskell/ghc-head/stack-test/.stack-work/docker/_home -e PATH=/opt/host/bin:/Users/callumr/Dropbox/Programming/Haskell/ghc-head/stack-test/.stack-work/docker/_home/.local/bin:/opt/ghc/head/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -v /Users/callumr/.stack:/Users/callumr/.stack -v /Users/callumr/Dropbox/Programming/Haskell/ghc-head/stack-test:/Users/callumr/Dropbox/Programming/Haskell/ghc-head/stack-test -v /Users/callumr/Dropbox/Programming/Haskell/ghc-head/stack-test/.stack-work/docker/_home:/Users/callumr/Dropbox/Programming/Haskell/ghc-head/stack-test/.stack-work/docker/_home -w /Users/callumr/Dropbox/Programming/Haskell/ghc-head/stack-test -v /Users/callumr/.stack/programs/x86_64-linux/stack-0.1.10.0/stack:/opt/host/bin/stack -t -i 2be8010a53ad /opt/host/bin/stack --internal-re-exec-version=0.1.10.0 --internal-docker-entrypoint "DockerEntrypoint {deUidGid = Nothing}" --skip-ghc-check -v build @(stack_15maE5vLiHaHnQnC7u59lN:System.Process.Read src/System/Process/Read.hs:273:3)
2015-12-13 17:36:38.710701: [debug] Run process: /usr/local/bin/docker start -a -i adb4895b9b44a55e7338970f5b675e5bb38e34d75cf0155e2fe534afb8c2c035 @(stack_15maE5vLiHaHnQnC7u59lN:System.Process.Run src/System/Process/Run.hs:105:5)
Version 0.1.10.0, Git revision a25892f8374d4bcbed6dd0f0c3bcb6cb93df490c (2778 commits) x86_64
2015-12-13 17:23:11.854239: [warn] Warning: /Users/callumr/.stack/config.yaml: Unrecognized field in ConfigMonoid: You can provide them in /Users/callumr/.stack/config.yaml, like this @(stack_CmMNd04E6niF0lFGLmWFKR:Data.Aeson.Extended src/Data/Aeson/Extended.hs:106:18)
2015-12-13 17:23:11.857598: [debug] Checking for project config at: /Users/callumr/Dropbox/Programming/Haskell/ghc-head/stack-test/stack.yaml @(stack_CmMNd04E6niF0lFGLmWFKR:Stack.Config src/Stack/Config.hs:576:9)
2015-12-13 17:23:11.858222: [debug] Loading project config file stack.yaml @(stack_CmMNd04E6niF0lFGLmWFKR:Stack.Config src/Stack/Config.hs:599:13)
2015-12-13 17:23:11.870496: [debug] Run process: ldd /opt/host/bin/stack @(stack_CmMNd04E6niF0lFGLmWFKR:System.Process.Read src/System/Process/Read.hs:273:3)
2015-12-13 17:23:11.883037: [debug] Trying to decode /Users/callumr/.stack/build-plan-cache/x86_64-linux-dkb98455da5dd905b08a4ce6671795697f/lts-3.6.cache @(stack_CmMNd04E6niF0lFGLmWFKR:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:55:5)
2015-12-13 17:23:11.900791: [debug] Success decoding /Users/callumr/.stack/build-plan-cache/x86_64-linux-dkb98455da5dd905b08a4ce6671795697f/lts-3.6.cache @(stack_CmMNd04E6niF0lFGLmWFKR:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:64:13)
2015-12-13 17:23:11.901244: [debug] Trying to decode /Users/callumr/.stack/indices/Hackage/00-index.cache @(stack_CmMNd04E6niF0lFGLmWFKR:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:55:5)
2015-12-13 17:23:12.197805: [debug] Success decoding /Users/callumr/.stack/indices/Hackage/00-index.cache @(stack_CmMNd04E6niF0lFGLmWFKR:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:64:13)
2015-12-13 17:23:12.211072: [debug] Run process: ghc --info @(stack_CmMNd04E6niF0lFGLmWFKR:System.Process.Read src/System/Process/Read.hs:273:3)
2015-12-13 17:23:12.254148: [debug] Run process: ghc --numeric-version @(stack_CmMNd04E6niF0lFGLmWFKR:System.Process.Read src/System/Process/Read.hs:273:3)
2015-12-13 17:23:12.284080: [debug] Run process: ghc-pkg --no-user-package-db field --simple-output Cabal version @(stack_CmMNd04E6niF0lFGLmWFKR:System.Process.Read src/System/Process/Read.hs:273:3)
2015-12-13 17:23:12.306031: [debug] Run process: ghc-pkg --no-user-package-db list --global @(stack_CmMNd04E6niF0lFGLmWFKR:System.Process.Read src/System/Process/Read.hs:273:3)
2015-12-13 17:23:12.322338: [debug] Run process: locale -a @(stack_CmMNd04E6niF0lFGLmWFKR:System.Process.Read src/System/Process/Read.hs:273:3)
2015-12-13 17:23:12.332716: [debug] Checking resolver: lts-3.6 @(stack_CmMNd04E6niF0lFGLmWFKR:Stack.Build.Source src/Stack/Build/Source.hs:164:17)
2015-12-13 17:23:12.333298: [debug] Trying to decode /Users/callumr/.stack/build-plan-cache/x86_64-linux-dkb98455da5dd905b08a4ce6671795697f/lts-3.6.cache @(stack_CmMNd04E6niF0lFGLmWFKR:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:55:5)
2015-12-13 17:23:12.339671: [debug] Success decoding /Users/callumr/.stack/build-plan-cache/x86_64-linux-dkb98455da5dd905b08a4ce6671795697f/lts-3.6.cache @(stack_CmMNd04E6niF0lFGLmWFKR:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:64:13)
2015-12-13 17:23:12.351041: [debug] Run process: ghc-pkg --global --no-user-package-db dump --expand-pkgroot @(stack_CmMNd04E6niF0lFGLmWFKR:System.Process.Read src/System/Process/Read.hs:273:3)
Invalid package ID: "array-0.5.1.0 base-4.9.0.0 binary-0.7.5.0 bytestring-0.10.7.0"
2015-12-13 17:36:40.532695: [debug] Run process: docker rm -f adb4895b9b44a55e7338970f5b675e5bb38e34d75cf0155e2fe534afb8c2c035 @(stack_15maE5vLiHaHnQnC7u59lN:System.Process.Read src/System/Process/Read.hs:273:3)

From what I can tell, the command ghc-pkg --global --no-user-package-db dump --expand-pkgroot is returning dependency information in an unexpected format. The format for dependencies from has changed in GHC HEAD. In 7.10 they looked like this:

[host] $ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.1
[host] $ ghc-pkg --global --no-user-package-db dump --expand-pkgroot
// skip
name: Cabal
depends:
    array-0.5.1.0-e29cdbe82692341ebb7ce6e2798294f9
    base-4.8.0.0-9015e10d2b2b0f71f570c3f2bbe09c8a
    binary-0.7.3.0-a62094e790b5d081d05ddc6a6a834684
    bytestring-0.10.6.0-3ca2b1049f8b29ce68d75c9a9933b807
    containers-0.5.6.2-32200224f699e9d2dc94c21f18d84191
    deepseq-1.4.1.1-936d71d0dde291c29ce67bcf563f5865
    directory-1.2.2.0-8cfc8dc7ab593a7f4f0dd1982e3bd349
    filepath-1.4.0.0-40d643aa87258c186441a1f8f3e13ca6
    pretty-1.1.2.0-c76df1e63c0ff1fb7797a99a20c7ea2a
    process-1.2.3.0-6c595818388e1c5dc8ac16de5f8d1a82
    time-1.5.0.1-30e4eefb5d3a5e08ebb5da0f9ddc2471
    unix-2.7.1.0-4577fa7ea76bd90259281765a2dc02e6

In HEAD they now look like this:

[host] $ docker run -it crogers/ghc-head:2015-12-13
[container] $ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.11.20151213
[container] $ ghc-pkg --global --no-user-package-db dump --expand-pkgroot
// skip
name: Cabal
depends:
    array-0.5.1.0 base-4.9.0.0 binary-0.7.5.0 bytestring-0.10.7.0
    containers-0.5.6.3 deepseq-1.4.2.0 directory-1.2.5.0
    filepath-1.4.1.0 pretty-1.1.2.0 process-1.4.1.0 time-1.5.0.1
    unix-2.7.1.1

Here are the full outputs for 7.10 and HEAD.

It seems weird to me that this change has happened, especially since the hashes have also disappeared from the package ids. Is this a GHC bug or expected behaviour? Either way it would be good to fix this so people can play with nightlies (really to try the new kind equality stuff) - this is the last part preventing me making an automated build for GHC HEAD nightlies that's easy to use. It will also present itself as a problem when GHC 8.0 is released in the coming month or so.
@CRogers
Copy link
Contributor Author

CRogers commented Dec 13, 2015

@luigy points out in #haskell-stack that those hashes which have disappeared are used for caching, which may be sad if they turn out to have gone.

@CRogers
Copy link
Contributor Author

CRogers commented Dec 13, 2015

I've written a failing test case for this behaviour. A bit more stuck on how to fix it.

@CRogers CRogers mentioned this issue Dec 13, 2015
Merged
@CRogers
Copy link
Contributor Author

CRogers commented Dec 13, 2015

Now an actual PR: #1510

@luigy
Copy link
Contributor

luigy commented Dec 13, 2015

hmm, so it looks like cabal lib used by latest ghc-pkg has changed from using newlines to spaces for depends: output

this is handled by stack here

but I wonder if this was intentional? cc @23Skidoo @dan-t
(haskell/cabal#1998)

@23Skidoo
Copy link

especially since the hashes have also disappeared from the package ids.

I think this was due to one of @ezyang's changes.

@CRogers
Copy link
Contributor Author

CRogers commented Dec 13, 2015

It could be that it always has put spaces between the ids, up to a maximum line length, except with the hashes on the end it always exceeded the maximum line line length...

@ezyang
Copy link

ezyang commented Dec 13, 2015

@CRogers Yes, that would be my guess. Stack should assume whitespace separated.

The hash removal was intentional (hashes in general haven't gone away from IPIDs, but I removed them from GHC's boot libraries, on the basis that they are the "one and only" libraries of this type for any GHC install), but maybe I can add it back.

@CRogers
Copy link
Contributor Author

CRogers commented Dec 13, 2015

@ezyang I'd be interested in what the Stack devs say, but I think I would prefer consistency across IPIDs (either they all have hashes or non of them do).

mgsloan added a commit that referenced this issue Dec 14, 2015
@mgsloan
Handle space separated pids in ghc-pkg dump #1509
9ee38c8
@mgsloan
Copy link
Contributor

mgsloan commented Dec 14, 2015

In 9ee38c8 I've made stack accept space separated build dependencies, and it now passes the test added in @CRogers PR. Leaving this ticket open for the discussion of the other change to the depends format.

I lean towards preferring consistency as well. However, if we can be reasonably sure that anything that expects an IPID will also accept these boot library versions, then I suppose it's not necessary. For example, here's how ghc-7.10.2 behaves if I try to pass these in as package-ids:

mgsloan@computer:~/fpco/stack$ stack exec -- ghc -package-id base-4.8.1.0
<command line>: cannot satisfy -package-id base-4.8.1.0
    (use -v for more information)
mgsloan@computer:~/fpco/stack$ stack exec -- ghc -package-id base-4.8.1.0-4f7206fd964c629946bb89db72c80011
ghc: no input files
Usage: For basic information, try the `--help' option.

@ezyang
Copy link

ezyang commented Dec 14, 2015

GHC 8.0 definitely will accept -package-id base-4.9.0.0, and it will specifically refer to the boot version. If it didn't, we wouldn't be able to build GHC!

I looked at the code in GHC, though, and it seems like the reason we don't make some sort of random hash is partially because GHC's build system may still have some (latent) bugs in it. I'll try to take a closer look.

@CRogers
Copy link
Contributor Author

CRogers commented Dec 27, 2015

Fixed

@CRogers CRogers closed this as completed Dec 27, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ezyang @mgsloan @23Skidoo @CRogers @luigy