Automate setup of ghcjs

[mgsloan]

Rough outline of what needs to be done here:

• Create and upload binary distributions for GHCJS
• Modify Stack.Setup to know how to install these binary distributions. If they work the same as the normal GHC ones, then this change should be pretty minimal:
  • Add soptsUseGHCJS :: Bool to SetupOpts
  • Most spots that use the string "ghc" will likely need to instead use "ghcjs" when this bool is true.
  • When downloading, get the url from a new ghcjs section of stack-setup.yaml instead of the ghc section.
  • Probably more I can't anticipate
• Add URLs for the binary distributions to https://raw.githubusercontent.com/fpco/stackage-content/master/stack/stack-setup.yaml
  • NOTE: make sure that extending this won't cause older stack versions to fail

[wuzzeb]

Just to point out, ghcjs doesn't install anything like normal GHC so I think instead what should happen is that once ghcjs is released to hackage (which should hopefully happen soon!), it should just be added into stackage. At that point, stack itself can compile ghcjs. Just to see that this actually works, I just built ghcjs today using stack and the following stack.yaml

packages:
- location:
    git: git@github.com:ghcjs/ghcjs
    commit: f4d437c21e7ec7c929748222151cf95e8efae4e7
resolver: lts-3.2

This gives the ghcjs binary in $PWD/.stack-work/install/x86_64-linux/lts-3.2/7.10.2/bin which isn't too useful, but if ghcjs was in the stackage snapshot then the ghcjs binary would be put into $HOME/.stack/snapshots/x86_64-linux/lts-3.2/7.10.2/bin directory. Then when stack was asked to compile using ghcjs, it can just call the ghcjs binary from the .stack/snapshots directory.

That gets the ghcjs binary. Next, https://github.com/ghcjs/ghcjs-boot/tree/improved-base should be merged into stackage somehow. Indeed, ghcjs-boot is a mini-stackage, with fixed versions of dependencies and a few patches on top of those packages. That way, the given stackage snapshot versions of things like bytestring, aeson, etc. are used.

Finally, the ghcjs-boot exe (source at https://github.com/ghcjs/ghcjs/blob/improved-base/src-bin/Boot.hs) must have parts of it re-implemented in stack. What it does now is clone the ghcjs-boot tree, apply patches, and build using ghcjs all these packages. But this is exactly what stack can do once ghcjs-boot is merged into stackage somehow.

[mgsloan]

Hey, thanks for looking into this!

ghcjs doesn't install anything like normal GHC

I've discussed it with Luite a little bit at ICFP, and it's certainly possible to have a binary build of GHCJS which installs the same way as GHC. This is better than having a source distribution because it takes lots of time to build everything, and things can go wrong along the way. The new GHCJS instructions seem quite promising (just cabal install http://ghcjs.luite.com/improved-base.tar.gz; ghcjs-boot). For me, haskell-src-exts failed to install for some reason, but I haven't looked into it..

It may indeed make sense to have stack install ghcjs from source in the meantime, I'll discuss this further with Luite.

Indeed, ghcjs-boot is a mini-stackage, with fixed versions of dependencies and a few patches on top of those packages. That way, the given stackage snapshot versions of things like bytestring, aeson, etc. are used.

Right, down the road we will need a separate set of stackage snapshots for GHCJS. Until then all dependencies will need to be explicit via the extra-deps field.

[mgsloan]

@wuzzeb , good news! We're going with pretty much the approach you describe. At some later date, it'll hopefully be switched over to binary distributions. @snoyberg Convinced me that we shouldn't "let perfect be the enemy of good" here, which makes sense to me. Here's how it's going to work:

• Download a .tar.gz much like the one at http://ghcjs.luite.com/improved-base.tar.gz
• If a recent enough version of cabal isn't on PATH, then install one locally, from source (needed for ghcjs-boot)
• Build it, hopefully with stack, possibly with cabal-install
• Run ghcjs-boot with the appropriate flags

I've discussed this with Luite a bit, and we've concluded that these pre-release versions of GHCJS should have version numbers which include the build date in them. For example, ghcjs-0.1.0.20150917. Once we have a tarball for a release like this we can do the following:

• Potentially host the tarball? Downloading it from ghcjs.luite.com would also work - it's up to Luite. I noticed that it doesn't support https, but this isn't much of an issue since download's hash is checked.
• Add a GHCJS section to https://github.com/fpco/stackage-content/blob/master/stack/stack-setup-2.yaml
• Merge the ghcjs-setup branch to master (not yet created)

Until this bit happens, I'll probably just have the implementation on the ghcjs-setup branch use the improved-base tarball

[snoyberg]

I'd recommend using Github releases for hosting the tarballs, if possible. That should also save Luite some bandwidth ;).

[mgsloan]

Here's how things currently work:

• It downloads the tarball specified in stack-setup-2.yaml:

ghcjs:
    source:
        ghcjs-0.1.0_ghc-7.10.2:
            url: ...

• It installs GHCJS from source using the stack.yaml in the tarball
• It boots GHCJS in the environment of the stack.yaml, locally installing cabal-install if necessary

TODO:

• Before this can work out of the box, a source release of GHCJS needs to be made, and stack-setup-2.yaml needs to be updated with the URL of the tarball. The GHCJS release needs to include these two PRs I made recently: Make ghcjs-boot --clean work for fresh install ghcjs/ghcjs#406 Add a stack.yaml ghcjs/ghcjs#407 . I don't think this should block merging the code in, though, as it should just make "stack setup" try to do something, where before it would just error out.
• Test if it works on windows

Questions:

• Install Node? I'm leaning towards not doing this unless there's demand. The ghcjs boot process will complain if it isn't around, and the user can install it with their package management of choice.
• Is it fine if we end up installing a new version of cabal to one of the user's snapshots? Seems fine to me, but it does mean that when using exec with that snapshot, it will shadow the system install.
• Should the GHCJS package DB go inside ~/.stack/programs/...? Currently, ghcjs-boot places it in ~/.ghcjs/x86_64-linux-0.1.0-7.10.2/, and doesn't seem to have flags to change it. I can only think of one reason to not leave it the way it is: if it's stored in ~/.stack/programs/.., then fully removing the version of ghcjs and its global DB is just a matter of removing a few things from that directory.

[ocharles]

Isn't node required to build anything that uses Template Haskell?

On Wed, 23 Sep 2015 4:04 am Michael Sloan notifications@github.com wrote:

Here's how things currently work:

• It downloads the tarball specified in stack-setup-2.yaml:

ghcjs:
source:
ghcjs-0.1.0_ghc-7.10.2:
url: ...

It installs GHCJS from source using the stack.yaml in the tarball

It boots GHCJS in the environment of the stack.yaml, locally
installing cabal-install if necessary

TODO:

Before this can work out of the box, a source release of GHCJS needs
to be made, and [
https://github.com/fpco/stackage-content/blob/1a661da435bf0de8da2e331014a18b17ad293dd8/stack/stack-setup-2.yaml]
needs to be updated with the URL of the tarball. The GHCJS release needs to
include these two PRs I made recently: ghcjs/ghcjs#406
ghcjs/ghcjs#406 ghcjs/ghcjs#407
ghcjs/ghcjs#407 . I don't think this should
block merging the code in, though, as it should just make "stack setup" try
to do something, where before it would just error out.

Test if it works on windows

Questions:

Install Node? I'm leaning towards not doing this unless there's
demand. The ghcjs boot process will complain if it isn't around, and the
user can install it with their package management of choice.

Is it fine if we end up installing a new version of cabal to one of
the user's snapshots?

Should the GHCJS package DB go inside ~/.stack/programs/.../?
Currently it ends up in ~/.ghcjs/x86_64-linux-0.1.0-7.10.2/

—
Reply to this email directly or view it on GitHub
#749 (comment)
.

[jwhitley]

I'll argue for not installing node. As you say, users will have a variety of management options and opinions around it. Tools like ndenv make it straightforward to install and manage multiple versions of node, and to automatically select between them on a per-project basis.

OTOH, guarding against and failing vociferously if the provided version of node is known to be incompatible would be very useful.

[mgsloan]

Isn't node required to build anything that uses Template Haskell?

Yup, but it's even more essential than that - it's also used while building GHCJS projects, to run Setup.hs.

OTOH, guarding against and failing vociferously if the provided version of node is known to be incompatible would be very useful.

Yup, I think this is the responsibility of ghcjs-boot. One of the reasons I'm inclined towards not automatically installing node is that we'd need a good way to pick a version. One possibility would be to have each ghcjs version in stack-setup-2 also specify a node version, but this seems ugly.

[mgsloan]

stack setup support is merged! All that's left is to try it out on windows and rename the installGHCJSPosix function if it works ;)

Recommended usage looks like:

resolver: ghcjs-0.1.0.20150924_ghc-7.10.2
compiler-check: match-exact

This will install an old-base version of ghcjs. improved-base / master versions of ghcjs will have version 0.2.0. I can upload a source tarball for those if there's demand.

[CRogers]

Awesome; I tried it out and it seems to manage installing ghcjs but fails at installing cabal. I was running inside a dir with the recommended stack.yaml you suggested. I'm on OS X and have stack version Version 0.1.5.0, Git revision 4271b996f3f5c236e5b2e995c37328aec644f9bf X86_64.

$ stack setup
# ...
Copied executables to /Users/callumr/.stack/programs/x86_64-osx/ghcjs-0.1.0.20150924_ghc-7.10.2/bin:
- hsc2hs-ghcjs
- haddock-ghcjs
- ghcjs-run
- ghcjs-pkg
- ghcjs-boot
- ghcjs
Installed GHCJS.
No 'cabal' binary found for use with GHCJS.  Installing a local copy of 'cabal' from source.
Could not parse '/Users/callumr/.stack/stack.yaml':
AesonException "failed to parse field 'resolver': key \"resolver\" not present"
See https://github.com/commercialhaskell/stack/blob/master/doc/yaml_configuration.md.
Building cabal-install for use by ghcjs-boot ... Process exited with ExitFailure 1: /Users/callumr/.local/bin/stack build cabal-install

Any idea what went wrong?

[CRogers]

I added:

resolver: ghcjs-0.1.0.20150924_ghc-7.10.2
compiler-check: match-exact

to ~/.stack/stack.yaml but I then get this message:

$ stack setup
Warning: /Users/callumr/.stack/stack.yaml: Unrecognized field in ConfigMonoid: resolver
No 'cabal' binary found for use with GHCJS.  Installing a local copy of 'cabal' from source.
Warning: /Users/callumr/.stack/stack.yaml: Unrecognized field in ConfigMonoid: resolver
GHCJS does not yet have its boot packages installed.  Use "stack setup" to attempt to run ghcjs-boot.
Building cabal-install for use by ghcjs-boot ... Process exited with ExitFailure 1: /Users/callumr/.local/bin/stack build cabal-install

[mgsloan]

@CRogers Thanks for trying it out! Do you happen to have STACK_YAML set? I didn't consider that this might be set, and so my invocations of stack should pass --stack-yaml. It looks like it's trying to load your global config yaml as a build config, which shouldn't be happening (not to be confused with ./stack/global/stack.yaml, which is a build config. This will be clarified in the future: #969).

[mgsloan]

@CRogers I've pushed a change that uses --stack-yaml instead of implicitly getting stack.yaml due to the current working directory. Hopefully that will fix it for you.

[CRogers]

Cool, that seems to have let it continue (it's now booting ghcjs). Thanks!

[legrostdg]

It works great!
Then I tried to install reflex... and got back to dependency hell. Can't wait for a stackage distribution for ghcjs :-).
Thanks a lot!

[conklech]

Preliminary test on Windows is a failure.

stack.yaml contains:

resolver: ghcjs-0.1.0.20150924_ghc-7.10.2
compiler-check: match-exact

Results:

c:\Projects\ghcjs-test>stack setup
Preparing to install GHCJS to an isolated location.
This will not interfere with any system-level installation.
No sha1 found in metadata, download hash won't be checked.
Downloaded ghcjs-0.1.0.20150924_ghc-7.10.2.
The following executables are missing and must be installed: gzip, tar

A clean Windows installation lacks gzip and tar. The normal GHC stack environment now contains the MSYS2 suite, which provides gzip by default but, inexplicably, not tar.

[snoyberg]

tar is included in MSYS2 as well, I'm not sure why it's not being found. Investigating.

[snoyberg]

Now I see what @mgsloan meant in our call earlier. The MSYS2 environment is not being initialized at all for the GHCJS installation, which is why the executable isn't being found. I think I can fix that.

[snoyberg]

I take it back, tar really isn't in MSYS2, strangely enough.

@mgsloan what do you think about just using 7z to decompress the files on Windows, like we do for MSYS itself and GHC?

[mgsloan]

@snoyberg Sounds like the right approach to me!

[mgsloan]

@snoyberg I've pushed a change which should make it use 7zip for this on windows. Does it work?

[snoyberg]

Discussed in private chat, repeating here: the current code ungzips the file, but doesn't untar. It needs to call 7z twice like we do for GHC bindists.

[mgsloan]

I've fixed the unzip issue, but now it runs into the long paths issue - #1145

Since this is essentially done, tracking the remaining work for windows support in that issue.

[fizruk]

One of the latest changes broke GHCJS setup for me:

$ stack setup --stack-yaml stack-ghcjs.yaml
No 'cabal' binary found for use with GHCJS.  Installing a local copy of 'cabal' from source.
Building cabal-install for use by ghcjs-boot ...
/home/ubuntu/.stack/programs/x86_64-linux/ghcjs-0.2.0.20151001_ghc-7.10.2/src/stack.yaml: canonicalizePath: does not exist (No such file or directory)

stack setup --stack-yaml stack-ghcjs.yaml returned exit code 1

Building cabal-install for use by ghcjs-boot ... Process exited with ExitFailure 1: /home/ubuntu/.local/bin/stack --stack-yaml /home/ubuntu/.stack/programs/x86_64-linux/ghcjs-0.2.0.20151001_ghc-7.10.2/src/stack.yaml build cabal-install Action failed: stack setup --stack-yaml stack-ghcjs.yaml

My stack-ghcjs.yaml (packages and extra-deps omitted):

flags: {}

packages: ...

resolver: ghcjs-0.2.0.20151001_ghc-7.10.2
compiler-check: match-exact

setup-info:
  ghcjs:
    source:
      ghcjs-0.2.0.20151001_ghc-7.10.2:
        url: "https://github.com/fizruk/ghcjs/releases/download/v0.2.0.20151001/ghcjs-0.2.0.20151001.tar.gz"

extra-deps: ...

I am not sure if this deserves a separate issue.

[mgsloan]

@fizruk Ah, I think that can happen if it was built with an earlier version of master, but not yet booted. I cleaned up the sdist unpack directory (was /home/ubuntu/.stack/programs/x86_64-linux/ghcjs-0.2.0.20151001_ghc-7.10.2/src/ghcjs-0.2.0.20151001_ghc-7.10.2/stack.yaml now it's /home/ubuntu/.stack/programs/x86_64-linux/ghcjs-0.2.0.20151001_ghc-7.10.2/src/stack.yaml.

I've pushed a change which allows the boot process to use the old path. Does it work now?

[fizruk]

@mgsloan yeah, seems to work now! Thanks!

[CRogers]

@fizruk How did you make your release for the master branch release of ghcjs? I'm referring to the 10.9mb file here. I'm trying to do the same thing but with the ghcjsi branch, however I seem to have missed a step as the source alone has lib/cache/boot.tar etc all empty :/

[fizruk]

@CRogers cabal sdist or stack sdist (GHCJS has non-trivial Setup.hs). Then you just attach the output .tar.gz to the release on GitHub (or anywhere you want to put it).

[mgsloan]

You will also need to modify update_archives.sh, which is used by Setup.hs before sdist. You'll need to have it checkout the right versions of shims and boot: https://github.com/ghcjs/ghcjs/blob/master/utils/update_archives.sh#L155 . Since I imagine the ghcjsi branch is lagging behind master, you'll probably need to pick particular SHAs.

I know, this kindof sucks. It'd be nice to get all the boot / sdist stuff simplified and always doing the "right thing" without configuration.

[CRogers]

Yeah, I worked out to modify the stuff to checkout in update_archives.sh but can't find the correct commits. They all failing on patching Win32 :(

[3noch]

Are there plans to support stack projects that have both GHC and GHCJS packages in them (e.g. server and front-end packages)?

[mgsloan]

@3noch I've spent some time considering this in the context of adding support for using GHCJS with stackage snapshots. Here are my thoughts: #1190