Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support sub-library dependencies (rebased). #5839

Closed

Conversation

philderbeast
Copy link
Contributor

I brushed the dust off #5659, rebased it and tested it with plugins-for-blobs. Seems to work.

> ~/.local/bin/stack clean

> ~/.local/bin/stack build --stack-yaml=stack.blobs.yaml
ghc-tcplugins-trace-0.1.0.0: unregistering
simple-smt-0.9.7: unregistering (local file changes: CHANGES SimpleSMT.hs simple-smt.cabal)
Building all executables for `build-plugins-for-blobs' once.
  After a successful build of all of them, only specified executables will be rebuilt.
build-plugins-for-blobs> configure (exe)
build-plugins-for-blobs> Configuring build-plugins-for-blobs-0.1.0...
build-plugins-for-blobs> build (exe)
ghc-tcplugins-trace    > configure (lib)
build-plugins-for-blobs> Preprocessing executable 'build-plugins-for-blobs'
build-plugins-for-blobs> Building executable 'build-plugins-for-blobs'
build-plugins-for-blobs> [1 of 2] Compiling Main
build-plugins-for-blobs> [2 of 2] Compiling Paths_build_plugins_for_blobs
ghc-tcplugins-trace    > Configuring ghc-tcplugins-trace-0.1.0.0...
ghc-tcplugins-trace    > build (lib)
build-plugins-for-blobs> Linking ...
ghc-tcplugins-trace    > Preprocessing library for ghc-tcplugins-trace-0.1.0.0..
ghc-tcplugins-trace    > Building library for ghc-tcplugins-trace-0.1.0.0..
simple-smt             > configure (lib)
ghc-tcplugins-trace    > [1 of 3] Compiling Plugins.Print.Constraints
simple-smt             > Configuring simple-smt-0.9.7...
ghc-tcplugins-trace    > [2 of 3] Compiling Plugins.Print.Flags
build-plugins-for-blobs> copy/register
simple-smt             > build (lib)
ghc-tcplugins-trace    > [3 of 3] Compiling Plugins.Print
simple-smt             > Preprocessing library for simple-smt-0.9.7..
simple-smt             > Building library for simple-smt-0.9.7..
build-plugins-for-blobs> Installing executable build-plugins-for-blobs in /.../bin
simple-smt             > [1 of 1] Compiling SimpleSMT
ghc-tcplugins-trace    > copy/register       
ghc-tcplugins-trace    > Installing library in /.../ghc-tcplugins-trace-0.1.0.0-93mkViScQTnCEFLPvUPNaq
ghc-tcplugins-trace    > Registering library for ghc-tcplugins-trace-0.1.0.0..
simple-smt             > copy/register       
simple-smt             > Installing library in /.../simple-smt-0.9.7-8DwBBLgHf4hIU6gyB8LxP1
simple-smt             > Registering library for simple-smt-0.9.7..
plugins-for-blobs      > build (internal-lib)
plugins-for-blobs      > Preprocessing library 'uom-quantity' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Building library 'uom-quantity' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Preprocessing library 'uom-th' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Building library 'uom-th' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Preprocessing library 'uom-plugin' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Building library 'uom-plugin' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Preprocessing library 'uom-plugin-defs' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Building library 'uom-plugin-defs' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Preprocessing library 'thoralf-theory' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Building library 'thoralf-theory' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Preprocessing library 'thoralf-encode' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Building library 'thoralf-encode' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Preprocessing library 'thoralf-plugin' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Building library 'thoralf-plugin' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Preprocessing library 'thoralf-plugin-rows' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Building library 'thoralf-plugin-rows' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Preprocessing library 'thoralf-plugin-uom' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Building library 'thoralf-plugin-uom' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Preprocessing library 'thoralf-plugin-defs' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > Building library 'thoralf-plugin-defs' for plugins-for-blobs-0.1.0..
plugins-for-blobs      > copy/register
plugins-for-blobs      > Installing internal library uom-quantity in /.../...-GwyR1RVr6FMD7RWaiQvBGo-uom-quantity
plugins-for-blobs      > Installing internal library uom-th in /.../...-Koubdi1gMCXDu2C87CZ3Z4-uom-th
plugins-for-blobs      > Installing internal library uom-plugin in /.../...-77EKG0oSYdm48UYcIgwxR3-uom-plugin
plugins-for-blobs      > Installing internal library uom-plugin-defs in /.../...-GPTrBk2wMfqJXQhmvZNGkj-uom-plugin-defs
plugins-for-blobs      > Installing internal library thoralf-theory in /.../...-6ntTdFkroSXAZMAxhqUQTe-thoralf-theory
plugins-for-blobs      > Installing internal library thoralf-encode in /.../...-K7QPeh1ogHA1982GJ0Y16Q-thoralf-encode
plugins-for-blobs      > Installing internal library thoralf-plugin in /.../...-HCgwvrdNNoAI0RArVjfoj7-thoralf-plugin
plugins-for-blobs      > Installing internal library thoralf-plugin-rows in /.../...-FfDbHar9QY8KemgkTi62J2-thoralf-plugin-rows
plugins-for-blobs      > Installing internal library thoralf-plugin-uom in /.../...-H0W6bQI6P5c3VAUT5bu8Sh-thoralf-plugin-uom
plugins-for-blobs      > Installing internal library thoralf-plugin-defs in /.../...-KUKBMGJE6LMAGMELEdh8Bx-thoralf-plugin-defs
Completed 4 action(s).

@philderbeast
Copy link
Contributor Author

I have another much larger repo that I am testing this on and it fails because, even though I've built stack with Cabal-3.8.1.0, stack is still choosing to use Cabal-3.2.1.0 when building the setup.

[warn] Cabal-simple_mPHDZzAJ_3.2.1.0_ghc-8.10.7: Encountered missing or private
[warn] dependencies:

How do I get stack to go all-in on Cabal-3.8.1.0?

, cpCabalVersion :: !Version
-- ^ This is the version of Cabal that stack will use to compile Setup.hs files
-- in the build process.
--
-- Note that this is not necessarily the same version as the one that stack
-- depends on as a library and which is displayed when running
-- @stack ls dependencies | grep Cabal@ in the stack project.

@philderbeast philderbeast marked this pull request as draft August 29, 2022 16:42
@mpilgrem
Copy link
Member

@philderbeast, I think the cpCabalVersion is determined by the version of GHC you have selected. If you want Cabal-3.8.1.0, you'll want a resolver that specifies GHC 9.4.x. Currently, only ghc-9.4.1 and ghc-9.4.2 exist and you will have to specify all the dependencies (other than those that ship with GHC) as extra-deps. You will also likely have to set allow-newer: true.

@mpilgrem
Copy link
Member

I don't know if this is relevant for your pull request, but I am keen to move Stack on to GHC 9.4.x as soon as a Stackage resolver is available and covers Stack's dependencies. That is because recent changes in MSYS2 can be problematic for Windows users of GHC < 9.4.1, as explained in GHC issue #21111.

@philderbeast
Copy link
Contributor Author

The test of this pull request on plugins-for-blobs that works is with ghc-9.2.3:

resolver: nightly-2022-07-10
install-ghc: false
system-ghc: true
packages:
- simple-smt
- ghc-tcplugins-trace/ghc-tcplugins-trace
- build
- .
extra-deps:
- ghc-corroborate-1.0.0@sha256:60e580741703235dc81232c41b6a7b29b5cf5e337c92ef1745509e8abf22d2b5,3855

@philderbeast
Copy link
Contributor Author

I don't know if this is relevant for your pull request, but I am keen to move Stack on to GHC 9.4.x as soon as a Stackage resolver is available and covers Stack's dependencies.

I'm good with that.

I did a little more testing with plugins-for-blobs and found that sub-library dependencies work for ghc-9.* but not for ghc-8.10.7. I used this set of testing projects configured to pickup the system GHC that I installed with ghcup.

> tree stack-blobs
stack-blobs
├── ghc-8.10.7.yaml
├── ghc-8.10.7.yaml.lock
├── ghc-9.0.2.yaml
├── ghc-9.0.2.yaml.lock
├── ghc-9.2.3.yaml
├── ghc-9.2.3.yaml.lock
├── ghc-9.2.4.yaml
└── ghc-9.2.4.yaml.lock

@mpilgrem
Copy link
Member

I think the CI failures may be something to do with the module name clashes discussed here: https://hackage.haskell.org/package/Cabal-syntax-3.6.0.0.

@AlistairB
Copy link

AlistairB commented Sep 1, 2022

Looks like #5691

I believe if you constrain Cabal >=3.8.1.0 you will also need Cabal-syntax >=3.8.1.0 as Cabal 3.8.1.0 depends on Cabal-syntax (and the older version of Cabal-syntax was just a placeholder release).

@philderbeast
Copy link
Contributor Author

the cpCabalVersion is determined by the version of GHC you have selected

Quite so. The command that fails shows that Cabal-3.6.3.0 is being used with ghc-9.2.4:

> stack build -v
...
~/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_3.6.3.0_ghc-9.2.4
> ghc-pkg list
~/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/lib/package.conf.d
    Cabal-3.8.1.0
    Cabal-syntax-3.8.1.0
    array-0.5.4.0
    base-4.17.0.0
    binary-0.8.9.1
    bytestring-0.11.3.1
    containers-0.6.6
    deepseq-1.4.8.0
    directory-1.3.7.1
    exceptions-0.10.5
    filepath-1.4.2.2
    ghc-9.4.2
    ghc-bignum-1.3
    ghc-boot-9.4.2
    ghc-boot-th-9.4.2
    ghc-compact-0.1.0.0
    ghc-heap-9.4.2
    ghc-prim-0.9.0
    ghci-9.4.2
    haskeline-0.8.2
    hpc-0.6.1.0
    integer-gmp-1.1
    libiserv-9.4.2
    mtl-2.2.2
    parsec-3.1.15.0
    pretty-1.1.3.6
    process-1.6.15.0
    rts-1.0.2
    stm-2.5.1.0
    system-cxx-std-lib-1.0
    template-haskell-2.19.0.0
    terminfo-0.4.1.5
    text-2.0.1
    time-1.12.2
    transformers-0.5.6.2
    unix-2.7.3
    xhtml-3000.2.2.1
ghc-pkg list
~/.ghcup/ghc/9.2.4/lib/ghc-9.2.4/lib/package.conf.d
    Cabal-3.6.3.0
    array-0.5.4.0
    base-4.16.3.0
    binary-0.8.9.0
    bytestring-0.11.3.1
    containers-0.6.5.1
    deepseq-1.4.6.1
    directory-1.3.6.2
    exceptions-0.10.4
    filepath-1.4.2.2
    ghc-9.2.4
    ghc-bignum-1.2
    ghc-boot-9.2.4
    ghc-boot-th-9.2.4
    ghc-compact-0.1.0.0
    ghc-heap-9.2.4
    ghc-prim-0.8.0
    ghci-9.2.4
    haskeline-0.8.2
    hpc-0.6.1.0
    integer-gmp-1.1
    libiserv-9.2.4
    mtl-2.2.2
    parsec-3.1.15.0
    pretty-1.1.3.6
    process-1.6.13.2
    rts-1.0.2
    stm-2.5.0.2
    template-haskell-2.18.0.0
    terminfo-0.4.1.5
    text-1.2.5.0
    time-1.11.1.1
    transformers-0.5.6.2
    unix-2.7.2.2
    xhtml-3000.2.2.1
> ghc-pkg list
~/.ghcup/ghc/8.10.7/lib/ghc-8.10.7/package.conf.d
    Cabal-3.2.1.0
    array-0.5.4.0
    base-4.14.3.0
    binary-0.8.8.0
    bytestring-0.10.12.0
    containers-0.6.5.1
    deepseq-1.4.4.0
    directory-1.3.6.0
    exceptions-0.10.4
    filepath-1.4.2.1
    ghc-8.10.7
    ghc-boot-8.10.7
    ghc-boot-th-8.10.7
    ghc-compact-0.1.0.0
    ghc-heap-8.10.7
    ghc-prim-0.6.1
    ghci-8.10.7
    haskeline-0.8.2
    hpc-0.6.1.0
    integer-gmp-1.0.3.0
    libiserv-8.10.7
    mtl-2.2.2
    parsec-3.1.14.0
    pretty-1.1.3.6
    process-1.6.13.2
    rts-1.0.1
    stm-2.5.0.1
    template-haskell-2.16.0.0
    terminfo-0.4.1.4
    text-1.2.4.1
    time-1.9.3
    transformers-0.5.6.2
    unix-2.7.2.2
    xhtml-3000.2.2.1

@mpilgrem
Copy link
Member

@philderbeast, the master branch version of Stack now builds with a dependency on Cabal-3.8.1.0.

@mpilgrem
Copy link
Member

@philderbeast, do you anticipate returning to this pull request? It seemed that you were close to your goal.

@philderbeast
Copy link
Contributor Author

@philderbeast, do you anticipate returning to this pull request? It seemed that you were close to your goal.

Yes, I think I'll be able to get to it tomorrow.

@mpilgrem
Copy link
Member

@philderbeast, I would be happy to help with the conflicts with the current master branch (which has moved on, somewhat) if you granted permission to commit to the pull request's compare branch.

@philderbeast
Copy link
Contributor Author

@mpilgrem thanks for the offer to help with the merge.

I've cherry-picked the one commit, 9ddf6f1, that matters for adding sub-library dependencies on branch add/sublib-deps-redo and am starting to test it. We have the integration test cabal-sublibrary-dependency and I will test it with packages I have that use sub-library dependencies, plugins-for-blobs.cabal being the biggest example.

@mpilgrem
Copy link
Member

mpilgrem commented Jan 25, 2023

@philderbeast, the error in the integration test - files > Error: Dependency on unbuildable library 'sub' from subproject is thrown by the Cabal library, in Distribution.Backpack.ConfiguredComponent.toConfiguredComponent.

@mpilgrem
Copy link
Member

mpilgrem commented Jan 25, 2023

More information about the error in the integration test (extracts, from a local build):

❯ stack --verbose build --cabal-verbose

2023-01-25 23:04:41.389056: [debug] Run process within C:\Users\mikep\Documents\Code\GitHub\cabalism\stack\test\integration\tests\cabal-sublibrary-dependency\files\: C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_sDt42OhJ_3.8.1.0_ghc-9.4.4.exe
configure

--libdir=C:\Users\mikep\Documents\Code\GitHub\cabalism\stack\test\integration\tests\cabal-sublibrary-
dependency\files\.stack-work\install\9a534d30\lib
--dependency=base=base-4.17.0.0 
--dependency=subproject=subproject-0.1.0.0-8HslT6gFjqDJdYuBDkDWlE 
--dependency=subproject:sub=subproject-0.1.0.0-8HslT6gFjqDJdYuBDkDWlE

2023-01-25 23:04:42.230628: [info] files     > Using Parsec parser
2023-01-25 23:04:42.230628: [info] files     > Configuring files-0.1.0.0...
2023-01-25 23:04:42.230628: [info] files     > Dependency base ==4.17.0.0: using base-4.17.0.0
2023-01-25 23:04:42.230628: [info] files     > Dependency subproject:{subproject, sub} ==0.1.0.0: using subproject-0.1.0.0
2023-01-25 23:04:42.230628: [info] files     > Dependency subproject:{subproject, sub} ==0.1.0.0: using subproject-0.1.0.0
2023-01-25 23:04:42.791241: [info] files     > Source component graph: component lib
2023-01-25 23:04:42.791241: [warn] files     > CallStack (from HasCallStack):
2023-01-25 23:04:42.791241: [warn] files     >   withMetadata, called at libraries\Cabal\Cabal\src\Distribution\Simple\Utils.hs:370:14 in Cabal-3.8.1.0:Distribution.Simple.Utils
2023-01-25 23:04:42.791241: [warn] files     > Error: Dependency on unbuildable library 'sub' from subproject

Looking at:

  • the --dependency=subproject:sub=subproject-0.1.0.0-8HslT6gFjqDJdYuBDkDWlE option in the Cabal configure command; and
  • the contents of the .stack-work\install\<hash>\lib\<platform>\subproject-0.1.0.0-AbwRb2DC7KN3C805E2GtpW-sub directory, the dependency option in the Cabal command seems to be specifying the wrong thing.

image

Also:

> stack exec -- ghc-pkg list
...
C:\Users\mikep\Documents\Code\GitHub\cabalism\stack\test\integration\tests\cabal-sublibrary-dependency\files\.stack-work\install\9a534d30\pkgdb
    subproject-0.1.0.0
    (z-subproject-z-sub-0.1.0.0)
> stack exec -- ghc-pkg describe z-subproject-z-sub-0.1.0.0
name:                 z-subproject-z-sub
version:              0.1.0.0
package-name:         subproject
lib-name:             sub
visibility:           public
id:                   subproject-0.1.0.0-AbwRb2DC7KN3C805E2GtpW-sub
key:                  subproject-0.1.0.0-AbwRb2DC7KN3C805E2GtpW-sub
abi:                  5b448a64deb5aaac652831aed5195cbe
exposed-modules:      SubLib
...

@mpilgrem
Copy link
Member

mpilgrem commented Feb 4, 2023

@philderbeast, I've been thinking about the failing integration test. I do not yet have an answer, but it seems to me that the underlying problem is that the deps :: Map PackageIdentifier GhcPkgId, to which configureOptsNoDir is actually applied, does not contain information about the sub library package z-subproject-z-sub with id=subproject-0.1.0.0-AbwRb2DC7KN3C805E2GtpW-sub.

I am going to do two minor things in the interim:

  1. I find the long line find (\(PackageIdentifier pn _, _) -> unPackageName pn == takeWhile (/= ':') subname) (Map.toList deps) difficult to follow. I am going to refactor to avoid doing so much in a single line.
  2. I think it woud be better if packageSubLibDeps was represented by a set of MungedPackageName (rather than Text). At the moment, the needed information gets encoded as Text and then later decoded.

@philderbeast
Copy link
Contributor Author

philderbeast commented Feb 4, 2023

@mpilgrem if you'd like to try a project with a lot of sublibs then here are stack and cabal commands for working with plugins-for-blobs. At first glance both seem to work and I'm using 39a8253 for the stack build and ghc-9.2.5 for the cabal build.

$ cabal build all --enable-tests --project-file=cabal.blobs.project
Up to date

$ ~/.local/bin/stack build plugins-for-blobs --test --no-run-tests --stack-yaml=stack-blobs.yaml
plugins-for-blobs> Test running disabled by --no-run-tests flag.

The tests pass except for doctests that have come loose.

A z3 release is required for the compilation.

$ z3 --version
Z3 version 4.8.12 - 64 bit

During the build of plugins-for-blobs, I dump a lot of output to the console like the following and that is normal (helps debugging the conversion from GHC typechecking constraints into SMT-LIB expressions):

; Compiling UoM
; GIVENS (conversions)
;      [WD] hole{co_a5rZ} {1}:: One
;                               ~# (Base "s" /: Base "s") (CNonCanonical)
;  =>  (=
;    (
;      (as
;         const
;         (Array String Int))
;      0)
;    (
;      (_
;         map
;         (-
;            (Int Int)
;            Int))
;      (store base "s" one)
;      (store base "s" one)))

@mpilgrem
Copy link
Member

mpilgrem commented Feb 7, 2023

@philderbeast, I thought, locally, that I had a solution (to the failing integration test cabal-sublibrary-dependency) with 751d2a6, which is why I pushed it, but I seem to have gone backwards ...

I think the solution lies in a change to Stack.Build.ConstructPlan.addDep and perhaps in a change to Stack.Build.ConstructPlan.combineMap too. Before the subproject:sub dependency is installed, Stack has to know that the source code is organised under 'package' subproject, and after it is installed, Stack has to know that the installed thing (with its own ghc-pkg id) is organised under 'package' z-subproject-z-sub.

@philderbeast
Copy link
Contributor Author

philderbeast commented Feb 8, 2023

@mpilgrem I added a cabal.project to cabal-sublibrary-dependency as a way to check that this test project can be built.

@mpilgrem
Copy link
Member

I rebased on the current master branch.

@mpilgrem
Copy link
Member

mpilgrem commented Mar 5, 2023

@philderbeast, I am making little progress on this. I have, however, moved the parts that I think are sound into Stack's master branch, and rebased on master. What is left here (currently) is: (a) turning on the cabal-sublibrary-dependency integration test (which currently fails); (b) turning off the "SubLibrary dependency is not supported, this will almost certainly fail." warning (which is still apt while the integration test is failing); and (c) my (erronerous) tinkering with addDep and addPackageDeps, which does not solve the problem.

@mpilgrem
Copy link
Member

mpilgrem commented Dec 1, 2023

I am closing this pull request, as it has been overtaken by #6343, which is close to being merged.

@mpilgrem mpilgrem closed this Dec 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants