Reinstall of package breaks things

[yaxu]

If I change then install a package I'm developing, ghci can no longer use it, it seems because there are two installations with the same version number. Transcript below. This is with the a fresh install via ghcup under linux.

Also, I don't know why I now have to specify --lib when presumably the module knows it's a library. As a general comment, documentation for v2-install doesn't seem up to date and I can't find clear explanations of the changes and how they can help my project

alex@stanage:~/tmp$ git clone https://github.com/tidalcycles/tidal.git
Cloning into 'tidal'...
remote: Enumerating objects: 125, done.
remote: Counting objects: 100% (125/125), done.
remote: Compressing objects: 100% (76/76), done.
remote: Total 9909 (delta 47), reused 93 (delta 32), pack-reused 9784
Receiving objects: 100% (9909/9909), 6.65 MiB | 1.54 MiB/s, done.
Resolving deltas: 100% (5383/5383), done.
alex@stanage:~/tmp$ cd tidal
alex@stanage:~/tmp/tidal$ cabal install --lib
Wrote tarball sdist to
/home/alex/tmp/tidal/dist-newstyle/sdist/tidal-1.4.6.tar.gz
Wrote tarball sdist to
/home/alex/tmp/tidal/dist-newstyle/sdist/tidal-parse-0.0.1.tar.gz
Resolving dependencies...
Up to date
alex@stanage:~/tmp/tidal$ echo "" >> src/Sound/Tidal/Pattern.hs 
alex@stanage:~/tmp/tidal$ cabal install --lib
Wrote tarball sdist to
/home/alex/tmp/tidal/dist-newstyle/sdist/tidal-1.4.6.tar.gz
Wrote tarball sdist to
/home/alex/tmp/tidal/dist-newstyle/sdist/tidal-parse-0.0.1.tar.gz
Resolving dependencies...
Build profile: -w ghc-8.6.5 -O1
In order, the following will be built (use -v for more details):
 - tidal-1.4.6 (lib) (requires build)
Starting     tidal-1.4.6 (lib)
Building     tidal-1.4.6 (lib)
Installing   tidal-1.4.6 (lib)
Completed    tidal-1.4.6 (lib)
alex@stanage:~/tmp/tidal$ ghci
GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help
Loaded package environment from /home/alex/.ghc/x86_64-linux-8.6.5/environments/default
Prelude> import Sound.Tidal.Context 

<no location info>: error:
    Ambiguous module name ‘Sound.Tidal.Context’:
      it was found in multiple packages: tidal-1.4.6 tidal-1.4.6
Prelude> 
Leaving GHCi.
alex@stanage:~/tmp/tidal$ grep tidal ~/.ghc/x86_64-linux-8.6.5/environments/default 
package-id tidal-1.4.6-29102d7626eceae1acf25d4320134825bb64112fb11058d394e0d55078882cde
package-id tidal-1.4.6-53f34a7fc0ce460f98fce9b704f6ab422c96435b533e18602b8728ba5e16553e

[yaxu]

This bug is still present - I spend time helping people work around this on a daily basis.

[m-renaud]

Hey @yaxu, I should have some time in the next couple of days to investigate this. I'm not familiar with this part of the code (@phadej could you give me some code pointers?). It sounds like install should check the environment for an existing package with the same version (but different hash) and:

a) Fail installation instead of happily installing something which will result in an error later
b) Support overwriting the existing package (with a --force option)

Also, I don't know why I now have to specify --lib when presumably the module knows it's a library.

There's discussion on splitting install --lib into its own command since its different in many ways (see #6481 for discussion).

As a general comment, documentation for v2-install doesn't seem up to date...

Yeah...it's not 🙃 I'm planning to get around to updating the docs soon.

[ertyseidohl]

Hi - Just want to +1 this bug and ask if there's documentation anywhere about how to get around it.

What's most frustrating to me about this bug is that it relies on global state - I messed up my installations somewhere and reverting my git repo back to my last commit didn't fix things.

[m-renaud]

Hey @ertyseidohl I'm actually actively working on this now, I'm hoping to have a PR out for review this week.

In the meantime, the workaround is to manually edit the GHC env file and remove the entry (or entries) of the libraries you're re-installing before running the install command. For example, say you have previously installed your-library-that-you-want-to-reinstall to the default environment, you will currently have something that looks like this:

$ cat ~/.ghc/x86_64-linux-8.8.3/environments/default 
clear-package-db
global-package-db
package-db /home/mrenaud/.cabal/store/ghc-8.8.3/package.db
package-id ghc-8.8.3
...
<global packages>
...
package-id text-1.2.4.0
package-id your-library-that-you-want-to-reinstall-0.1.2.3-longhashvalue

If you manually edit this file to remove the entry for your-library-that-you-want-to-reinstall before running cabal install --lib then that should work just fine. Let me know if that's not the case.

[yaxu]

Thanks for looking at this. I've helped many people with broken cabal installs lately, and I'm also seeing this problem with duplicate versions for people who are installing straight from hackage, so presumably it's possible to install the same package version twice even if they have the same hash.
I'm also seeing people who have upgraded the tidal package but ghci is still loading the old one.
The only reliable remedy seems to be removing the environment file, and most reliably the whole of .ghc and .cabal, and installing with v1-install.
Just anecdotes, as I rarely have direct access to the machines.

[m-renaud]

@yaxu do you happen to have a repro case when installing directly from hackage? If the package has the same hash there shouldn't be any modifications made to the env file.

[m-renaud]

Also @yaxu, this is a bit of a meta-comment/question, but I just wanted to make sure that your use case actually requires modifying the global package environment; apologies if you're already aware of this and have determined it doesn't work for you.(this goes back to the v1->v2 behaviour change and workflows is not well documented).

With the v1- commands, in order to use a library in a package/project you're developing you would need to v1-install it to the global package DB and then you would have access to it in your project. With the v2- commands, cabal handles downloading and "installing" libraries into a local environment to your project is handled automatically, as long as you have them listed in the build-depends attribute. So for the ghci above, assuming you are in the tidal project you should just be able to run cabal repl :tidal and it will make all of the modules within the tidal library available to the repl.

[yaxu]

Thanks @m-renaud, I'd love to understand this more!

The use case I have is supporting a library (tidal) for end-user programmers, in this case musicians who are generally unfamiliar with Haskell and its build systems. The don't want to deal with git repos, or develop any code, they just want to install the tidal library according to the easiest and most reliable method, and use it in the ghci repl (via an editor plugin).

As I understand it there isn't really a global environment with v2- commands, just a 'default' one, is that right? So I assume any problem I'm having with the default package environment would still be a problem if I used another named environment.

[m-renaud]

Gotcha, thanks for the context. Yeah it sounds like using a global environment is probably the best way forward, and the root of the issue is when you make changes to tidal locally and re-install it corrupts the env file. You should soon be able to run:

cabal install --lib --force --package-env=ENV

which would overwrite any existing packages referenced in ENV with the newer versions.

[phadej]

I’m sorry to interrupt but I veto any cli changes to cabal install —lib interface until there is a clear plan for the final UI. I have opened various issues and proposed various solutions. The problem is not implementing the changes, but to agree how package environment management should look like.

That said —force is a bad argument name for "don’t consider current environment contents”.

[yaxu]

Hi @phadej, if the UI isn't finalised, to the extent that bugs like this aren't able to be fixed, why is this already in production? I've found the transition to v2-install a bumpy ride to say the least. I've been spending a hours a week recently helping people with install problems.

[phadej]

That is a good question. Yet adding more debt and implementing not so well-thought out ideas is kind of the way we got here in the first place. install —lib doesn’t seem to be designed at all. Just implemented. (I haven’t looked into history how it came to existence, will do tomorrow).

…

On 28. Apr 2020, at 22.14, Alex McLean ***@***.***> wrote:  Hi @phadej, if the UI isn't finalised, to the extent that bugs like this aren't able to be fixed, why is this already in production? I've found the transition to v2-install a bumpy ride to say the least. I've been spending a hours a week recently helping people with install problems. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[m-renaud]

I've got a pretty decent handle on the install --lib code now, I was actually consider doing a pretty substantial prefactor/refactoring before making the change since everything is pretty tangled together at the moment. If desired I can send out a PR for that.

I can start with that and then we can start sketching out what a cabal env API would look like (I have some ideas, where's the best place to put those?).

[m-renaud]

Also, is there a good real-time place to chat about this? #cabal irc? Going back and fourth on comments works for some things but getting on the same page is sometimes easier and more expedient with an "in-person" discussion.

[ertyseidohl]

If it helps, I've written up my process with all of the errors and fixes at https://organicdonut.com/2020/04/cabal-package-installation-woes/

[m-renaud]

Hey @ertyseidohl I just took a look at your post and it doesn't appear that you need to use cabal install at all. If you are developing an executable that depends on libraries from hackage, you don't need to install the libraries, as long as your .cabal file declares the necessary build-depends all you need to do to build and run your executable is run cabal run :server.

Do you have the full code that I could look at to confirm? Also, what version of cabal are you using?

[ertyseidohl]

I think part of my problem is just being new to haskell and not understanding what I'm doing with the package management :)

My code (at the time of writing) is here: https://github.com/ertyseidohl/crossword-hs/tree/072b4efa11346dc12b20835d09d4eb7559c13a36

[m-renaud]

Ahh yeah there was a time when you needed to install libraries to the global package namespace to use them but that hasn't been the case for quite some time (although there are random docs scattered across the internet that still say you need to do that). The best resource for getting started is the Getting Started guide in the user docs. If you have any issues following those instructions please file an issue about it :)

[yaxu]

You should soon be able to run:

cabal install --lib --force --package-env=ENV

which would overwrite any existing packages referenced in ENV with the newer versions.

Returning to this, and trying the --force option.. It doesn't help.

alex@toaster:~$ cabal install --package-env=forcetest --lib tidal-1.7.10
Resolving dependencies...
Up to date
alex@toaster:~$ cabal install --package-env=forcetest --lib tidal-1.8.0
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] next goal: tidal (user goal)
[__0] rejecting: tidal-1.8.0 (constraint from user target requires ==1.7.10)
[__0] rejecting: tidal-1.7.10, tidal-1.7.9, tidal-1.7.8, tidal-1.7.7,
tidal-1.7.6, tidal-1.7.5, tidal-1.7.4, tidal-1.7.3, tidal-1.7.2, tidal-1.7.1,
tidal-1.7, tidal-1.6.1, tidal-1.6.0, tidal-1.5.2, tidal-1.5.1, tidal-1.5.0,
tidal-1.4.9, tidal-1.4.8, tidal-1.4.7, tidal-1.4.6, tidal-1.4.5, tidal-1.4.4,
tidal-1.4.3, tidal-1.4.2, tidal-1.4.1, tidal-1.4.0, tidal-1.3.0, tidal-1.2.1,
tidal-1.2.0, tidal-1.1.2, tidal-1.1.1, tidal-1.1.0, tidal-1.0.14,
tidal-1.0.13, tidal-1.0.12, tidal-1.0.11, tidal-1.0.10, tidal-1.0.9,
tidal-1.0.8, tidal-1.0.7, tidal-1.0.6, tidal-1.0.5, tidal-1.0.4, tidal-1.0.3,
tidal-1.0.2, tidal-1.0.1, tidal-1.0.0, tidal-0.9.10, tidal-0.9.9, tidal-0.9.8,
tidal-0.9.7, tidal-0.9.6, tidal-0.9.5, tidal-0.9.4, tidal-0.9.3, tidal-0.9.2,
tidal-0.9.1, tidal-0.9, tidal-0.8.2, tidal-0.8.1, tidal-0.8, tidal-0.7.1,
tidal-0.7, tidal-0.6, tidal-0.5.3, tidal-0.5.2, tidal-0.5.1, tidal-0.5,
tidal-0.4.36, tidal-0.4.35, tidal-0.4.34, tidal-0.4.33, tidal-0.4.32,
tidal-0.4.31, tidal-0.4.30, tidal-0.4.29.1, tidal-0.4.29, tidal-0.4.28,
tidal-0.4.27, tidal-0.4.26, tidal-0.4.24, tidal-0.4.23, tidal-0.4.21,
tidal-0.4.20, tidal-0.4.19, tidal-0.4.17, tidal-0.4.16, tidal-0.4.15,
tidal-0.4.14, tidal-0.4.13, tidal-0.4.12, tidal-0.4.11, tidal-0.4.10,
tidal-0.4.9, tidal-0.4.8, tidal-0.4.7, tidal-0.4.6, tidal-0.4.5, tidal-0.4.4,
tidal-0.4.3, tidal-0.4.2, tidal-0.4.1, tidal-0.4, tidal-0.3.9, tidal-0.3.8,
tidal-0.3.7, tidal-0.3.6, tidal-0.3.5, tidal-0.3.4, tidal-0.3.3, tidal-0.3.2,
tidal-0.3.1, tidal-0.3, tidal-0.2.13, tidal-0.2.11, tidal-0.2.10, tidal-0.2.9,
tidal-0.2.7, tidal-0.2.6, tidal-0.2.4, tidal-0.2.3, tidal-0.2.2.8,
tidal-0.2.2.7, tidal-0.2.2.6, tidal-0.2.2, tidal-0.2.1, tidal-0.2,
tidal-0.1.0.1, tidal-0.1 (constraint from user target requires ==1.8.0)
[__0] fail (backjumping, conflict set: tidal)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: tidal

alex@toaster:~$ cabal install --force --package-env=forcetest --lib tidal-1.8.0
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] next goal: tidal (user goal)
[__0] rejecting: tidal-1.8.0 (constraint from user target requires ==1.7.10)
[__0] rejecting: tidal-1.7.10, tidal-1.7.9, tidal-1.7.8, tidal-1.7.7,
tidal-1.7.6, tidal-1.7.5, tidal-1.7.4, tidal-1.7.3, tidal-1.7.2, tidal-1.7.1,
tidal-1.7, tidal-1.6.1, tidal-1.6.0, tidal-1.5.2, tidal-1.5.1, tidal-1.5.0,
tidal-1.4.9, tidal-1.4.8, tidal-1.4.7, tidal-1.4.6, tidal-1.4.5, tidal-1.4.4,
tidal-1.4.3, tidal-1.4.2, tidal-1.4.1, tidal-1.4.0, tidal-1.3.0, tidal-1.2.1,
tidal-1.2.0, tidal-1.1.2, tidal-1.1.1, tidal-1.1.0, tidal-1.0.14,
tidal-1.0.13, tidal-1.0.12, tidal-1.0.11, tidal-1.0.10, tidal-1.0.9,
tidal-1.0.8, tidal-1.0.7, tidal-1.0.6, tidal-1.0.5, tidal-1.0.4, tidal-1.0.3,
tidal-1.0.2, tidal-1.0.1, tidal-1.0.0, tidal-0.9.10, tidal-0.9.9, tidal-0.9.8,
tidal-0.9.7, tidal-0.9.6, tidal-0.9.5, tidal-0.9.4, tidal-0.9.3, tidal-0.9.2,
tidal-0.9.1, tidal-0.9, tidal-0.8.2, tidal-0.8.1, tidal-0.8, tidal-0.7.1,
tidal-0.7, tidal-0.6, tidal-0.5.3, tidal-0.5.2, tidal-0.5.1, tidal-0.5,
tidal-0.4.36, tidal-0.4.35, tidal-0.4.34, tidal-0.4.33, tidal-0.4.32,
tidal-0.4.31, tidal-0.4.30, tidal-0.4.29.1, tidal-0.4.29, tidal-0.4.28,
tidal-0.4.27, tidal-0.4.26, tidal-0.4.24, tidal-0.4.23, tidal-0.4.21,
tidal-0.4.20, tidal-0.4.19, tidal-0.4.17, tidal-0.4.16, tidal-0.4.15,
tidal-0.4.14, tidal-0.4.13, tidal-0.4.12, tidal-0.4.11, tidal-0.4.10,
tidal-0.4.9, tidal-0.4.8, tidal-0.4.7, tidal-0.4.6, tidal-0.4.5, tidal-0.4.4,
tidal-0.4.3, tidal-0.4.2, tidal-0.4.1, tidal-0.4, tidal-0.3.9, tidal-0.3.8,
tidal-0.3.7, tidal-0.3.6, tidal-0.3.5, tidal-0.3.4, tidal-0.3.3, tidal-0.3.2,
tidal-0.3.1, tidal-0.3, tidal-0.2.13, tidal-0.2.11, tidal-0.2.10, tidal-0.2.9,
tidal-0.2.7, tidal-0.2.6, tidal-0.2.4, tidal-0.2.3, tidal-0.2.2.8,
tidal-0.2.2.7, tidal-0.2.2.6, tidal-0.2.2, tidal-0.2.1, tidal-0.2,
tidal-0.1.0.1, tidal-0.1 (constraint from user target requires ==1.8.0)
[__0] fail (backjumping, conflict set: tidal)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: tidal

alex@toaster:~$ 

[Mikolaj]

Returning to this, and trying the --force option.. It doesn't help.

Perhaps it has a stale .ghc.environment* somewhere?

[ulysses4ever]

I'm sorry but was --force ever merged? I thought it never graduated from a design proposal.

[Mikolaj]

Doh, I guess, it's accepted only because it's a prefix of --force-reinstalls and we silently accept prefixes of commands. Too bad.