Continually blocked on upstream packages

[AlistairB]

To update to a new ghc / cabal version they need to be available in Debian first. Unfortunately there is a lot of lag from when the new versions are released to when they appear as debian packages. This seems to be due to @hvr being the sole person responsible for the packaging, but often not being available to do this work.

This is not a criticism of @hvr who has given a tremendous amount to the Haskell ecosystem ❤️ , but it is just a reality that there is consistent lag here.

Options

Try to improve the situation around debian package updates

We could chase this up with https://github.com/haskell-CI/haskell-ci which is the home of the debian/ubuntu packages. We could also pull in the Haskell foundation who may have ideas.

Switch to ghcup

Ghcup does not depend on these same debian packages and does promptly add new versions. I've been playing with what this might looking like.

Pros

• New versions of cabal, ghc and stack are promptly added.
• Once mechanism for installing cabal, ghc and stack.
• Removes some complexity of the installation from these images.
• Ghcup is further consolidated as the one true way (excluding stack) of installing ghc / cabal. Haskell github actions are also using ghcup. Actually, it seems it uses the hvr ubuntu PPA where possible, then falls back to ghcup?
• Ghcup has (preliminary?) support for windows, so it provides a path to a windows docker haskell image.
• I believe ghcup has ARM support, so could help support arm.
• We could support more linux distros that ghcup supports if we want ie. alpine

Cons

• Ghcup is not a great fit for docker. Ghcup is a tool to manage many versions of tools, when in docker we only want to provide one exact version. Thus it is additional bloat + complexity when it shouldn't in theory be required. For this reason I believe no other official programming language images are using an equivalent tool.
• Ghcup installs for the current user, where as in docker we want to install for all users. For example a common pattern in docker is to switch to a new user with less privileges. However, this new user cannot use ghc / cabal / stack that were installed for a different user by ghcup. There may be ways to work around this, potentially requiring ghcup changes.

Conclusion

???

Not sure, ghcup is a bit of a poor fit philosophically, however may be a great pragmatic choice resulting in a net win. If we do not go with ghcup, I think we should attempt to improve the lag problem with the current debian packages.

@psftw @hasufell any thoughts?

[bgamari]

For what it's worth, ghcup is just a (very good!) tool for installing and managing standard GHC release artifacts. I have considered adding Debian packaging jobs to GHC's release pipeline. If it would be helpful for you I would be happy to give it a try; I don't think it should be difficult.

[AlistairB]

For what it's worth, ghcup is just a (very good!) tool for installing and managing standard GHC release artifacts. I have considered adding Debian packaging jobs to GHC's release pipeline. If it would be helpful for you I would be happy to give it a try; I don't think it should be difficult.

Yes that would be super useful and would eliminate the lag problem in regards to ghc 😄

Docker haskell also includes cabal, but perhaps that can be solved as part of the cabal release process.

The other consideration might be ubuntu packaging, which the haskell github action uses. The action also uses GHC + cabal installed on the default GHA agents.. not sure what they depend on. @jared-w may know more.

[hasufell]

Ghcup is not a great fit for docker.

I believe it is. Many already use it in docker and you only install what you need. I don't believe it's bloat. And it makes a lot of sense for users who want to add additional GHC versions to an existing image.

Ghcup installs for the current user, where as in docker we want to install for all users.

Since the latest version, GHCup can install into any directory via https://gitlab.haskell.org/haskell/ghcup-hs#isolated-installs

This still requires (for GHC) that the target directory is empty, but we may lift that restriction, which means you'll be able to do ghcup install ghc latest --isolate /usr, which will put ghc binaries in /usr/bin/ghc.

So for now you could install via ghcup install ghc latest --isolate /opt/ghc and put /opt/ghc/bin in PATH.

[AlistairB]

I believe it is. Many already use it in docker and you only install what you need. I don't believe it's bloat.

I guess to be more specific it is not part of the core offering of the image. I think even if we use ghcup to install ghc etc. we still ideally want to strip it out from the final image. This is because docker is provides a mechanism to switch ghc versions with docker tags, so ghcup shouldn't be needed by the end user.

Having said that I think a haskell:ghcup image would be useful if people want to use that as a base to get an exact combination of ghc, cabal + stack versions.

So for now you could install via ghcup install ghc latest --isolate /opt/ghc and put /opt/ghc/bin in PATH.

Thank you! Your example image does exactly what I was failing to do 😅

I have adapted your example further to get more towards what I was trying to do. That being ghcup is used to install ghc, cabal + stack. However, a multi stage docker image is used to only provide an image with ghc, cabal and stack only without ghcup or other cache used as part of building.

What do you think?

EDIT: I think with these improvements I am more strongly in favour of the ghcup based solution.

[hasufell]

so ghcup shouldn't be needed by the end user

I'd recommend to install it by default. I'd argue it's a common use case to derive images and augment them, e.g. when you have multi-GHC builds or simply need to do local experiments.

The binary is, compared to the 2GB GHC install, very small.

What do you think?

Yeah

[AlistairB]

I'd recommend to install it by default. I'd argue it's a common use case to derive images and augment them, e.g. when you have multi-GHC builds or simply need to do local experiments.

The binary is, compared to the 2GB GHC install, very small.

It's not a size issue, it is more that it doesn't really fit with how docker works and could be a foot gun. Using ghcup in docker and mutating the container can have unusual effects.

I think for specific use case wanting multiple ghc versions in an image I would recommend a haskell:ghcup image (which I think can be a follow up issue if we go with ghcup). Or in many cases you would just use the haskell docker image for each version you needed to build that step.

---

One other concern I have noticed is the ghcup based image is a bit larger. Not a deal breaker but not ideal. I stripped out /opt/ghc/share/ which helped a bit (this seems to be just docs). After that the size diff is:

Ghcup based 8.10.4 - 2.34GB
Current 8.10.4 - 1.52GB

I wonder if there is an easy way to strip out something unnecessary from /opt/ghc/lib?

[hasufell]

It's not a size issue, it is more that it doesn't really fit with how docker works and could be a foot gun. Using ghcup in docker and mutating the container can have unusual effects.

Unusual effects?

This is about:

1. deriving docker images based on an existing image, so you only have to install one additional GHC version instead of two from scratch. This is for Dockerfile developers, they know what they do.
2. being able to shell into the container and quickly experiment with other GHC versions

[AlistairB]

Unusual effects?

I would prefer not to get bogged down on this finer point. I think this can be discussed further if ghcup is chosen as the best way forward. Cheers

[hazelweakly]

The other consideration might be ubuntu packaging, which the haskell github action uses. The action also uses GHC + cabal installed on the default GHA agents.. not sure what they depend on. @jared-w may know more.

To first address the Ubuntu packaging; the Ubuntu packaging does not seem to be updated anymore because it was also done by @hvr. As such, the upstream GitHub preinstalled software switched from the PPA to using ghcup by default and the GitHub actions followed suit. Any pre-installed version of GHC or cabal on a GitHub runner is currently installed by ghcup (or chocolatey on windows)

I think, currently, the ideal situation to install GHC in a docker container would be to use ghcup with the isolate flag and manually copy over all the binaries individually in a subsequent stage. This lets the Dockerfile be self-documenting in which Haskell components are installed as well as minimizing the size and following standard docker conventions

[AlistairB]

Thanks for the info @jared-w .

I think, currently, the ideal situation to install GHC in a docker container would be to use ghcup with the isolate flag and manually copy over all the binaries individually in a subsequent stage. This lets the Dockerfile be self-documenting in which Haskell components are installed as well as minimizing the size and following standard docker conventions

Agreed. This is pretty much what is sketched in #44

---

I'll see if I can understand why the ghcup installed ghc takes up more space compared to the current debian packaging.

Also @psftw would need to chime in before we could consider proceeding.

[AlistairB]

So it seems that ghcup is installing the base libraries with profiling enabled/supported if I understand correctly. The current debian packaging splits out the profiling versions into a separate package ie. ghc-8.10.4-prof so we don't get those in the existing images.

I stripped out the profiling related files and now the sizes are comparable.

@hasufell would a flag to install / not install profiling support in the base libraries be something ghcup might look to support?

Another question, is there currently a way to subscribe to ghcup releases? This would be useful around knowing to bump ghcup in the images.

[hasufell]

So it seems that ghcup is installing the base libraries with profiling enabled/supported if I understand correctly. The current debian packaging splits out the profiling versions into a separate package ie. ghc-8.10.4-prof so we don't get those currently.

I stripped out the profiling related files and now the sizes are comparable.

@hasufell would a flag to install / not install profiling support in the base libraries be something ghcup might look to support?

I'm not sure there's a clean UX for this, since ghcup after all isn't a real package manager, but an installer. Re-installing those profiling libs would require a full reinstall of the GHC version. If the build system (./configure) had a flag to omit installing profiling libs, however, that could easily be supported ad-hoc.

But if the only use case is docker, manually removing files may be the way to go.

Another question, is there currently a way to subscribe to ghcup releases? This would be useful around knowing to bump ghcup in the images.

There's an atom feed https://gitlab.haskell.org/haskell/ghcup-hs/-/tags

But new tags don't always mean new releases. ghcup list is the source of truth.

[AlistairB]

I've spent some time polishing up the changes for this in #44 . More feedback is welcome :)

But if the only use case is docker, manually removing files may be the way to go.

No worries, happy to go with the stripping approach for now.

But new tags don't always mean new releases. ghcup list is the source of truth.

No problem.

---

@hasufell now that the PR has progressed well and hopefully can be merged soon once @psftw gives his thoughts, perhaps we can discuss the question of whether to leave ghcup in the final image. So I'm pretty strongly against this. Reasons..

Additional bloat

• There are additional packages that ghcup requires that we aren't otherwise required in the final image.
• The pretty negligible 23MB size of the ghcup executable.

I'm not suggesting this is significant, but it is a downside.

Additional releases

If we don't support ghcup, I would only bump ghcup when it might have a substantial impact to ghc / cabal / stack in the images. (Or when we bump something else in an image)

If we officially support ghcup, then we should bump ghcup everytime it gets a new release IMO. This means the images need to be updated more (seems to be on average once a month).

• More effort required by docker-haskell maintainers.
• More updating of the images by users who I would suggest rarely want to use ghcup in the images.

Mutating a running image

So mutating a running container will not be persisted by default. So if your goal is to play around with ghcup, you will need to re setup everything if you exit the container. There are then interesting things like docker commit which can persist changes made into a new image.

All of this is finicky and non-obvious. I think there are many people who are new to docker who could get fooled by this. In other words I consider this a foot gun.

We actually currently disable stack from installing new GHC versions, so it is something of a current assumption around the images.

Upsides of having ghcup in the final image?

So my suggestion is that having ghcup is almost always not actually useful in the image. The core reason is that docker itself is providing a mechanism to access different versions of ghc.

deriving docker images based on an existing image, so you only have to install one additional GHC version instead of two from scratch. This is for Dockerfile developers, they know what they do.

I would suggest that in almost all cases if you need multiple ghc versions, you would use those versions in different docker images. For example perhaps you want to test your library builds on different ghc verisons. In that case you want to build based on multiple docker images for each version in parallel. You do not have any need to have multiple ghc versions loaded in a single image.

being able to shell into the container and quickly experiment with other GHC versions

Again, I think it much simpler to use docker for this. Most developers are familiar with switching versions in docker. As opposed to learning a new tool. As mentioned above, any mutation of the running container is not easy to persist, so you would need to repeat it.

Conclusion

To sum up, I can see multiple downside and I cannot think of a case where using ghcup in the image is useful.

I think the key requirement the current images don't really cover is when you want specific GHC(s) with specific cabal and specific stack. I see a haskell:ghcup image that doesn't include GHC / cabal / stack by default as the solution to this. (I would have this be a separate issue / future change.)

[hasufell]

There are additional packages that ghcup requires that we aren't otherwise required in the final image.

That's not true. ghcup is a static executable that requires only curl at runtime and so does cabal.

So mutating a running container will not be persisted by default. So if your goal is to play around with ghcup, you will need to re setup everything if you exit the container. There are then interesting things like docker commit which can persist changes made into a new image.

Sorry, I don't understand what you're trying to say. If people don't understand docker, they should read the documentation.

Again, I think it much simpler to use docker for this. Most developers are familiar with switching versions in docker. As opposed to learning a new tool.

Can you tell me then how to augment an existing image, that I just derived from haskell/docker, with a new additional GHC version in one line?

Either you have to replicate all the ghcup foo in your Dockerfile or figure out what files you need to copy from an existing image layer. The latter is also inflexible, because it doesn't work in a running container.

---

At any rate, I'm not an active user of these images. I prefer to download alpine docker images of minimal size and then perform ghcup installation inside of them. That saves me from downloading several GBs of unpacked GHC.

The time it takes to unpack/install GHC usually is much less than downloading a huge image.