Cannot install ihaskell-display widgets with stack

[vaibhavsagar]

I've built the ihaskell-display/* modules and installed ihaskell with stack, but I can't use any widget functionality and my attempts to import IHaskell.Display.Widgets are unsuccessful (Failed to load interface for ‘IHaskell.Display.Widgets’). Is there something simple I'm missing?

[schernichkin]

Thank you for closing my issue, now I'm going to close yours)

I did closer examination of how Stack works and why we have such behavior (it was an scenario in my issue which can be stably reproduced on current repo).

tl;dr; To fix your must place stack.yaml to the folder with notebooks (or any parent folder). Stack.yaml should be like this:

flags:
  ihaskell:
    binpkgdb: false
packages:
- location:
    git: https://github.com/gibiansky/IHaskell.git
    commit: f39b812fdcc5566210f9960f8218c5fc7fd40a77
  subdirs:
  - .
  - ./ipython-kernel
  - ./ghc-parser
  - ./ihaskell-display/ihaskell-aeson
  - ./ihaskell-display/ihaskell-blaze
  - ./ihaskell-display/ihaskell-charts
  - ./ihaskell-display/ihaskell-diagrams
  - ./ihaskell-display/ihaskell-gnuplot
  - ./ihaskell-display/ihaskell-hatex
  - ./ihaskell-display/ihaskell-juicypixels
  - ./ihaskell-display/ihaskell-magic
  - ./ihaskell-display/ihaskell-plot
  - ./ihaskell-display/ihaskell-static-canvas
  - ./ihaskell-display/ihaskell-widgets
extra-deps: []
resolver: lts-9.0

Or just run from the IHaskell directory. In this case everything under ihaskell-display/* will not work, because each directory contains it's own stack.yaml created under assumption that all other libraries are already installed, which is not true. Everything under other directories will work.

Let's turn to details. Stack maintains it environment by taking packages of specific GHC version (defined by resolver), then installed packages related to specific version of Stackage database, then custom packages, defined in stack.yaml and installed to project-local database (under .stack-work folder). Three statements can help clue the things together:

• Stack will not install to local Stackage cache anything what not belonging to the Stackage build plan. It's possible to have less installed packages than in Stackage database, but not more, neither different versions. When your build your project Stackage packages get installed to the local Stackage cache (if they are not already there) and become available to GHC. If you have not build your project yet they probably not yet there.
• All additional libraries, specified in stack.yaml installed to project-local cache. They will work as they were a part of Stackage build plan, but only for the current project, completely isolated from any other projects.
• Stack looking for stack.yaml from the current to the parent folder, once it find it, it will use it. If it will not find it, it will use global-project configuration. Hence stack.yaml in child folder will completely override any parent folder's settings.
• (one more) stack install command may be misleading. Unlike cabal-install Stack do not have user package database (with the exception of global-project which is not very useful though). Thus stack build will perform an actual installation and it will only affect Stackage cache (and will only install what belongs to it) and project-local database. And stack install will just copy binaries.

Now back to IHaskell. To make IHaskell works all your need is working Haskell environment, IHaskell binary and Jupyter of course. So, if your just issue stack install your probably will not notice any problem. But display modules get loaded dynamically using current environment. So, lets check environments:

stack exec -- ghc-pkg list under IHaskell folder (after building the project):

/home/username/.stack/programs/x86_64-linux/ghc-8.0.2/lib/ghc-8.0.2/package.conf.d
    Cabal-1.24.2.0
    array-0.5.1.1
    base-4.9.1.0
    binary-0.8.3.0
... more lines ...
/home/username/.stack/snapshots/x86_64-linux/lts-9.0/8.0.2/pkgdb
    Boolean-0.2.4
    Chart-1.8.2
    Chart-cairo-1.8.2
    HTTP-4000.3.7
... more lines ...
/home/username/Projects/IHaskell/.stack-work/install/x86_64-linux//8.0.2/pkgdb
    ghc-parser-0.1.8.0
    ihaskell-0.8.4.0
    ihaskell-aeson-0.3.0.0
    ihaskell-blaze-0.3.0.0
... more lines ...

As you can see, we have three section here: ghc-8.0.2 packages, Stackage database packages and IHaskell local packages. All IHaskell modules installed to project local database, IHaskell packages not included to lts-9.0, so issuing same command from different folder will return same first two sections, but without IHaskell packages:

/home/username/.stack/programs/x86_64-linux/ghc-8.0.2/lib/ghc-8.0.2/package.conf.d
    Cabal-1.24.2.0
    array-0.5.1.1
    base-4.9.1.0
    binary-0.8.3.0
... more lines ...
/home/username/.stack/snapshots/x86_64-linux/lts-9.0/8.0.2/pkgdb
    Boolean-0.2.4
    Chart-1.8.2
    Chart-cairo-1.8.2
    HTTP-4000.3.7
... more lines ...
/home/username/.stack/global-project/.stack-work/install/x86_64-linux/lts-9.0/8.0.2/pkgdb
    (no packages)

Running stack exec -- ghc-pkg list under ihaskell-widgets will yield similar results

  ...........
/home/username/Projects/IHaskell/ihaskell-display/ihaskell-widgets/.stack-work/install/x86_64-linux/lts-9.0/8.0.2/pkgdb
    (no packages)

This is because ihaskell-widgets has it's own stack.yaml, which does not include ihaskell modules and they also not in lts-9.0. So, if your enter Examples folder of ihaskell-widgets and try to run notebook, IHaskell will create kernel instance with Stack. Stack will search stack.yaml (or, possibly, just working directory, I'm not sure in exact behavior) an will find one of ihaskell-widgets. Which does not contain any IHaskell packages. Thus, attempt to import any IHaskell module will fail.

[schernichkin]

Got few more thoughts about this problem. I think there is a still bug in IHaskell. First, each of ihaskell-display/* package “claims” that it can be built with lts-9.0 along. This is wrong, because lts-9.0 does not include ihaskell. Second, regardless it may seem non-essential issue, it really affects newcomers. People want to check-out the code, compile it and click over examples to get an overall idea. If code does not work, no one will spend his time to sort things out. Most peple will just say “ok, huskell sucks, it does not work”. So, it’s important to make it work out of the box.

[vaibhavsagar]

I agree that this is something worth fixing. One thing I looked into was installing the display modules into Stack's global-project under ~/.stack/global-project, but that didn't seem to make them available to IHaskell. Is this something you tried as well?

[schernichkin]

I've found 2 problem preventing from running notebooks from ihaskell-display/* folders.

1. stack.yaml in each folder which hides root stack.yaml. It will lead to Failed to load interface for ‘IHaskell.Display.* error message. To fix it just delete stack.yaml (is there any good reason to have stack.yaml in ihaskell-display/* folders?)
2. In some packages (e.g. ihaskell-charts) notebook located directly in root folder. In this case ghci will notice module source file will not start kernel with following message: attempting to use module 'IHaskell.Display.Charts’ (./IHaskell/Display/Charts.hs) which is not loaded. This can be solved by moving notebook to sub-folder.

[vaibhavsagar]

Thanks for looking into it 😄. It sounds like you're focused on the problem of running the display notebooks from the source repository itself, and I would like a solution where the display modules are available to all IHaskell notebooks I create. However, if you'd like to submit a pull request with your changes that would be welcome.

[schernichkin]

To make display modules work, you should use stack.yaml I've posted in my first message. It can be either in notebook folder or in global-project folder (in the latter case make sure there is no another stack.yaml, which will override global-project). Then just install it by stack build command. Compiler will warn No extra-dep setting found for package at URL: https://github.com/gibiansky/IHaskell.git but this exactly what we want. Then start IHaskell, all display modules, specified in stack.yaml will be available.

[vaibhavsagar]

Hmm, in that case, it might be better to use the Nix method, because that makes the display modules available even in the presence of a stack.yaml and everything else. If I could fix the bug that prevents hot-loading from working (#728) I might recommend using that instead.

[defjaf]

@schernichkin I'm not sure I understand the prescription. Can the folder just be an arbitrary folder or do I need to make a new stack project (with stack new or similar)? Am I then meant to be installing the notebooks themselves?

@vaibhavsagar This may be related to the fact that I don't really understand the "Where are my packages" section of the docs... Would it be possible to add a little more detail (or a little less detail but a little more of a cookbook!)? To be more specific, if I have a package that I would usually install (or have already installed) via cabal so that ghc sees them, what are the instructions?

[schernichkin]

@defjaf stack new will basically create stack.yaml and some other files described in new-template.hsfiles. You may use stack newbut you don't have to. Creating stack.yaml with correctly filled packages section will be enough. You may put notebooks to root folder (where stack.yaml located) or to any subfolder.

[stschiff]

Hi, after reading through your comments on this issue, I think I have found an elegant solution: I have pretty much copied the contents from the stack.yaml file that @schernichkin's kindly supplied here into my global stack.yaml file under ~/.stack/global/stack.yaml. You can then go to any directory that is not a stack project (for example, say, your Desktop), and simply run stack install ihaskell, then ihaskell install, then stack install ihaskell-diagrams and stack install ihaskell-charts and whatever other display package you'd like to have installed. After having done that, I find that running ihaskell notebooks anywhere (using stack exec jupyter -- notebook) will work just fine with all the display instances that you've installed.

@schernichkin: I don't understand why you say "the global stack project is pretty useless". I think for this kind of setup, where you want some libraries installed to be visible everywhere, the global stack project is exactly what we want.

As a side note, on my Mac I found this blog post to be very useful: You can basically set up ihaskell to even work without invoking stack exec jupyter..., by simply adding the stack exec -- part to the kernel description file, as described in that post.

[stschiff]

If you can reproduce this approach using the global stack.yaml project, perhaps this should be put into the docs. It's certainly quite a bad design aspect that in order to run any notebook with correct display, you'd have to place it into the ihaskell code directory in order for it to work.

[vaibhavsagar]

I'm satisfied with the approach of putting the ihaskell-* modules in my extra-deps for each project.

[jamesdbrock]

Hi, after reading through your comments on this issue, I think I have found an elegant solution

That's a very elegant solution, thank you @stschiff . We should incorporate that solution into the IHaskell/Dockerfile.

[vaibhavsagar]

@jamesdbrock opening an IHaskell notebook in the ihaskell directory in the Docker container will correctly load all working display modules, is there a reason you can't use this approach?

[jamesdbrock]

I don't have an ihaskell directory in my container. But more generally, I want to be able to save .ipynb files in directories organized by project rather than application, as one does. Stack global default project works really well for that.

Opening JupyterLab notebooks in a Stack project directory with a custom stack.yaml also work well, as long as the resolver in the project stack.yaml agrees with the IHaskell installation.

[jamesdbrock]

Here is a complete Docker Stacks image which uses @stschiff 's ideas about using the stack global project to install IHaskell in such a way that notebooks can be stored in and launched from any directory, no stack.yaml needed.

https://github.com/jamesdbrock/ihaskell-notebook