For discussion: some significant changes to MinGHC

[snoyberg]

I started off yesterday playing with the idea of bundling Git with MinGHC. A bunch of changes in that direction are on the git branch. Unfortunately going down this path ended up raising a lot of tangential issues, so I'd like to raise them here for discussion.

• Currently our msys is based on the older msys1, while most people are moving to msys2 (related: Use MSYS2 instead of MSYS1 #20 Can't build GHC from Git with MinGHC #40). It seems that PortableGit includes msys2 (which is how I ended up with the weird network issue today).
  • One option is to include PortableGit and put its msys on the PATH. However, this won't expose the pacman installer, which some people may want.
  • Another option is to include msys2 itself, but that would mean (1) we wouldn't get Git, and would have to maintain our own bundle, since msys2 is only available via an installer.
  • And then of course we have option 3 of sticking with msys1. It's worth pointing out that msys1 is significantly smaller in size than the other two (they both take ~40MB compressed).
• PortableGit is distributed as a 7z archive. It's easy to move around the 7z.exe and 7z.dll files for decompression. On the git branch, I took the approach of bundling the .7z archives in the installer itself, together with 7z.exe and 7z.dll, and then decompressing from the nsis script. This makes the executable creation process faster, and reduces dependencies on the build system (no need for bzip2, gzip, or bsdtar).
• Another change I made to simplify setup was to move to http-client for doing the downloads instead of shelling out to wget. This change is completely orthogonal to others, and I don't feel strongly about it at all (85bfde5).
• Permissions issues are a problem (Better story for admin vs non-admin #2). With PortableGit, we have to install to a user-writable place so that the /tmp directory is writable. I solved this by installing everything into $HOME/AppData/Roaming/minghc-7.10.1-i386. I'm leaning towards this being a good idea as opposed to just a hack to get PortableGit working.
• Issue Support Stackage better #19 requested better Stackage support, which means bundling the new stackage-cli tools. That, combined with cabal, alex, happy, and things we add in the future, will lead to a lot of things to keep track of. Instead, I set things up with a single .7z archive containing all of the executables, which seems like an easier thing to manage.

Sorry for the large, all-at-once braindump. I thought I'd be able to play with just Git, but I ended up unraveling many other things.

[ndmitchell]

• I think moving to msys2 in some form is inevitable - sticking to old versions of unsupported tools isn't really feasible. As such, the size is kinda irrelevant. Whether we grab pacman or not is something I don't mind too much.
• Grabbing msys2 from the official place (even if that is install and then zip up ourselves) seems better than sourcing it from PortableGit. Often you can unzip installers.
• Using http-client seems fine now. Until now, we had a bootstrapping problem in that installing network was a real pain. Now everyone uses MinGHC so it's not such an issue.
• Originally we tried to source binaries from the sources, but then it became clear we basically had to compile them ourselves, so one 7z seems sensible now. One option would be checking the binaries into git directly, and then we'd want a separate 7z per executable to minimize the diffs.

[snoyberg]

Forgot to mention: installer for this available at: https://s3.amazonaws.com/download.fpcomplete.com/minghc/minghc-7.10.1-i386-git.exe

[ndmitchell]

Btw, what's the motivation for bundling git? Installing it separately has never been that onerous for me... One possible line of attack would be that MinGHC could include enough to compile GHC itself, which would be a nice line in the sand as to what we do/do not include.

[snoyberg]

Some of the Stackage tooling provides faster and more secure package index downloading by using Git. It's an optional dependency, but it will make users' lives better if present.

[ndmitchell]

Ah, that makes sense then.

[snoyberg]

Your points about storing the executables in the repository are interesting. That could definitely simplify things: we would have a directory with a collection of .7z files, and just extract all of them to $INSTDIR/bin, am I correct?

[ndmitchell]

Thats roughly what I'm thinking - seems a bit easier than having half on amazon and half the stuff in git, plus that way I can update the pieces on my own.

[snoyberg]

The one other thing I forgot to mention is that I'll need to manually unpack the bindist from GHC HQ and repack it as a 7z (or automate that process, either way). So we'll end up with I believe three sources for building this:

• Git, containing: 7z.exe and 7z.dll, and a bunch of .7z archives
• GHC, stored on S3 as a 7z
• PortableGit, either downloaded from S3 or Github (I lean towards S3 so that they can't pull the rug out from under us in the future)

If you're OK with this, I'll move ahead with the rest of the changes for the git branch and get it ready for a PR

[ndmitchell]

GHC ships the bindist as an xz, can't 7zip open that? I thought it was 7zip under the hood.

Other than that, all sounds good to me. Although we won't want to roll it out until network gets released, I guess.

[snoyberg]

Right you are, I didn't realize 7z could handle that without extra DLLs. Cool, the makes my life much simpler (and the download a bit smaller too apparently).

[snoyberg]

And you're right, we'll need to wait for the network release before doing this.

[DanBurton]

Apparently 7zip can handle tons of formats, including xz, bz2, and gz. Neat.

Supported formats:

• Packing / unpacking: 7z, XZ, BZIP2, GZIP, TAR, ZIP and WIM
• Unpacking only: ARJ, CAB, CHM, CPIO, CramFS, DEB, DMG, FAT, HFS, ISO, LZH, LZMA, MBR, MSI, NSIS, NTFS, RAR, RPM, SquashFS, UDF, VHD, WIM, XAR and Z.

http://www.7-zip.org/

[snoyberg]

OK, I think I've done enough damage on the git branch for one day. Diff at:

https://github.com/fpco/minghc/compare/git?expand=1

I'm sure the Shake/nsis parts of this will make you cry @ndmitchell ;). I've also generated 4 installers (7.8.4/7.10.1 32/64 bit) and am uploading them to a temporary location on S3:

• https://s3.amazonaws.com/download.fpcomplete.com/minghc/withgit/minghc-7.8.4-i386.exe
• https://s3.amazonaws.com/download.fpcomplete.com/minghc/withgit/minghc-7.10.1-i386.exe
• https://s3.amazonaws.com/download.fpcomplete.com/minghc/withgit/minghc-7.8.4-x86_64.exe
• https://s3.amazonaws.com/download.fpcomplete.com/minghc/withgit/minghc-7.10.1-x86_64.exe

The first two are available now, the other two should be up in ~10 minutes (slow uploads right now for some reason). I've been using this, and it seems to be OK. Things to consider changing:

• I guess we should change the order of the PATH now so that the $APPDATA/cabal/bin comes first, correct?
• Instead of putting the git directory inside of the minghc directory, we could consider putting it in $APPDATA directly so that you don't get two installs of Git with two installs of MinGHC
• I'm using the 32-bit PortableGit for both installers, though I suppose we could use the 64-bit instead
• An alternative directory structure would be to put all of the directories of subdirectories of $APPDATA\MinGHC

On that note, I think I'll finally have some dinner :)

[akhra]

If everything's going in Roaming, IMO put it each part in its own directory
there, not gathered under MinGHC. Cabal and GHC already have their own
directories there no matter what, so it actually feels like the more
consistent organizational scheme.

On Wed, May 6, 2015 at 3:27 PM, Michael Snoyman notifications@github.com
wrote:

OK, I think I've done enough damage on the git branch for one day. Diff at:

https://github.com/fpco/minghc/compare/git?expand=1

I'm sure the Shake/nsis parts of this will make you cry @ndmitchell
https://github.com/ndmitchell ;). I've also generated 4 installers
(7.8.4/7.10.1 32/64 bit) and am uploading them to a temporary location on
S3:

https://s3.amazonaws.com/download.fpcomplete.com/minghc/withgit/minghc-7.8.4-i386.exe

https://s3.amazonaws.com/download.fpcomplete.com/minghc/withgit/minghc-7.10.1-i386.exe

https://s3.amazonaws.com/download.fpcomplete.com/minghc/withgit/minghc-7.8.4-x86_64.exe

https://s3.amazonaws.com/download.fpcomplete.com/minghc/withgit/minghc-7.10.1-x86_64.exe

The first two are available now, the other two should be up in ~10 minutes
(slow uploads right now for some reason). I've been using this, and it
seems to be OK. Things to consider changing:

• I guess we should change the order of the PATH now so that the
  $APPDATA/cabal/bin comes first, correct?
• Instead of putting the git directory inside of the minghc
  directory, we could consider putting it in $APPDATA directly so that you
  don't get two installs of Git with two installs of MinGHC
• I'm using the 32-bit PortableGit for both installers, though I
  suppose we could use the 64-bit instead
• An alternative directory structure would be to put all of the
  directories of subdirectories of $APPDATA\MinGHC

On that note, I think I'll finally have some dinner :)

—
Reply to this email directly or view it on GitHub
#43 (comment).

[snoyberg]

@tejon One downside is that the uninstallers would no longer be able to get rid of PortableGit, since other installation may be using that. But maybe that's a good thing. I do lean towards doing this, so unless @ndmitchell is opposed we can move foward.

[ndmitchell]

Putting installs in $APPDATA feels wrong - it's not where I expect programs to be, and goes against the traditional Windows setup. I know Cabal gets installed in $APPDATA, but in a way that is data, as it was compiled on the users system. If we want to install PortableGit shared, I would have thought somewhere in Program Files was the right approach - e.g. MinGHC\7.10.1 for the version specific bits, and MinGHC\PortableGit for the other stuff.

[snoyberg]

I agree that AppData is not correct, but program files isn't an option due
to the permissions issues. I can't imagine we're the first people to need
to install an application for just a single user, surely there must be some
windows standard for this, right?

On Thu, May 7, 2015, 8:50 AM Neil Mitchell notifications@github.com wrote:

Putting installs in $APPDATA feels wrong - it's not where I expect
programs to be, and goes against the traditional Windows setup. I know
Cabal gets installed in $APPDATA, but in a way that is data, as it was
compiled on the users system. If we want to install PortableGit shared, I
would have thought somewhere in Program Files was the right approach - e.g.
MinGHC\7.10.1 for the version specific bits, and MinGHC\PortableGit for the
other stuff.

—
Reply to this email directly or view it on GitHub
#43 (comment).

[snoyberg]

Via StackOverflow I came to some Microsoft docs, which looks like a good place to install to (%LOCALAPPDATA%\Programs).

[snoyberg]

Johan released s new version of network, so we can release whenever we're ready.

[snoyberg]

I've updated the install to use $LOCALAPPDATA\Programs. I'm not longer certain about putting git in a separate directory. The reason being: consider if the user decided that the installation directory should be c:\minghc. Should we install Git in c:\git or c:\minghc\git? The former seems strange. I suppose we could prompt the user to choose an installation directory for Git as well, but that seems overkill.

I'd rather not let that restructuring block the rest of this, so @tejon, if you want to continue the discussion on that, could you open it as a separate issue?

@ndmitchell I'm going to close this issue and open a PR. Once this is merged to master, I'll regenerate all of the installers. I'll likely upload to a temporary location and ask for some testing before making it the official installer.

[ndmitchell]

I see no reason to share git - we don't share anything else. Why not just install lots of copies of Git inside each MinGHC? You are trading off disk-space (which is basically free and small in comparison to the size of GHC) for robustness (which we all like).

[mboes]

I like the idea of keeping to having minghc being a self contained
"environment". It provides just GHC, packages and build tools (such as
Cabal), and assumes no particular external dependencies. If the build tools
require something internally, such as Git, then that can be included, but
isolated from the user.

If the user wants Git, then the user should install Git and manage herself
the Git installation that she uses, independent of the git installation
that MinGHC uses internally. That way MinGHC uses a known good git setup,
while the user is still free to use a completely different Git setup
without interference.

As Neil says, the cost is of course a bit of extra disk space, but that
sounds fair in this day and age.

[akhra]

I'm not longer certain about putting git in a separate directory. The reason being: consider if the user decided that the installation directory should be c:\minghc. Should we install Git in c:\git or c:\minghc\git?

I thought we were talking about a mandatory location akin to Cabal's, not just a new default. If it's still under user control then I agree, keep it gathered.

[mantkiew]

While I understand the need for MSYS2, I don't quite understand the need for git. Many people use the GitHub desktop tool, for example, which comes with it's own PortableGit, which gets updated everytime GitHub updates itself. And many people may already have git in their PATH, so messing up with that (i.e., adding git which comes with MinGHC into user's path) may cause surprises. But if Git is needed by MinGHC, then probably it's safer for it to be somewhat "hidden" and isolated from user's environment.

[3noch]

I miss stackage-purge.

[snoyberg]

The need for git is that some of the tools we're bundling- stackage-update in particular- use it. It doesn't work to have git hidden away, because then the tools won't be able to find it themselves. Perhaps the right solution here is to have an option in the installer as to whether to include the git.exe directory on the PATH or not.

@3noch Let's follow up on that with #19.

[elieux]

Some follow-ups on Git for Windows MSYS2 vs. vanilla MSYS2:

• I'm not sure what "portable" means here. First, it's not true that only installers are available. There's a .tar.xz for every MSYS2 release (and 7-zip was proven to be able to unpack that). Also, MSYS2 is portable by design, there's no separate portable version.
• MSYS2 currently doesn't feature a mingw Git package, but it should get there once Git for Windows gets a stable release.

However that doesn't mean that GfW isn't better suited for MinGHC. For example, the writable /tmp requirement will probably be lifted soon in GfW (see git-for-windows/git#143), but inclusion of this feature in vanilla MSYS2 is not certain. There may be more such patches in GfW.

By the way, should #20 now be closed?

[snoyberg]

I was unaware of the .tar.xz version, that's what I was looking for. The writable /tmp would also be very nice. But getting a package which includes git and pacman seems ideal.

On that note: I think #20 should remain open until we should pacman.

[elieux]

GfW provides two environments:

• the Git & MSYS2 essentials for users that only want Git (this is what MinGHC is using)
• the SDK, which is more of a MSYS2 distro with the GfW patches and package repositories included, for people who want to use the MSYS2 part of GfW for building software and such

Personally, I think this could work for MinGHC, too, but it requires maintaining a pacman repository with GHC (and cabal-install and others).