Please, solve the Cabal build dependency design issue between the projects

[Anton-Latukha]

hnix-store-core & hnix-store-remote separately cabal v2-build fine.

The HLS founds the cradle in hnix-store home directory, as it should - because cabal.project:

packages: ./hnix-store-core/*.cabal ./hnix-store-remote/*.cabal

Is put there.

This directly ties the two project dependencies together - and now HLS does not work:

Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: hnix-store-core-0.3.0.1 (user goal)
[__1] trying: cryptohash-sha512-0.11.100.1 (dependency of hnix-store-core)
[__2] next goal: base (dependency of hnix-store-core)
[__2] rejecting: base-4.14.1.0/installed-4.14.1.0 (conflict: cryptohash-sha512
=> base>=4.5 && <4.14)
[__2] skipping: base-4.14.1.0, base-4.14.0.0 (has the same characteristics
that caused the previous version to fail: excluded by constraint '>=4.5 &&
<4.14' from 'cryptohash-sha512')
[__2] rejecting: base-4.13.0.0, base-4.12.0.0, base-4.11.1.0, base-4.11.0.0,
base-4.10.1.0, base-4.10.0.0, base-4.9.1.0, base-4.9.0.0, base-4.8.2.0,
base-4.8.1.0, base-4.8.0.0, base-4.7.0.2, base-4.7.0.1, base-4.7.0.0,
base-4.6.0.1, base-4.6.0.0, base-4.5.1.0, base-4.5.0.0, base-4.4.1.0,
base-4.4.0.0, base-4.3.1.0, base-4.3.0.0, base-4.2.0.2, base-4.2.0.1,
base-4.2.0.0, base-4.1.0.0, base-4.0.0.0, base-3.0.3.2, base-3.0.3.1
(constraint from non-upgradeable package requires installed instance)
[__2] fail (backjumping, conflict set: base, cryptohash-sha512,
hnix-store-core)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: base, hnix-store-core,
cryptohash-sha512

If one works on the dependency migration in the hnix-store-core - IDE stops to work, which is convenient.

Tying the projects with cabal.project this way would always complicate more than solve because it goes against the design of the Haskell ecosystem.

---

Nota bene: I had Nix infrastructures at work, participated in Nix, participated in Nixpkgs, formed/created the haskell-with-nixpkgs and test-haskell-nixpkgs-integration-action.

---

Projects in monorepo can do/pick 1 of 2 things:

1. Sequentially release projects and make changes propagate classically: Make a release of hnix-store-core with some changes, then provide support for new release in hnix-store-remote and make new release of it, then hnix takes the releases and provides support for them and makes a release. CI works simple and great in this case, because dependencies are taken from the Hackage/Stackage or the descendent of their stores, Nixpkgs, and CI build support is tracked through package versions support.

2. Move everything at the same time in the monorepo. It is so convenient to make changes everywhere at once. Did changes in hnix-store-core then hnix-store-remote, etc. At that point there is no concern to support Hackage. Especially if projects cross-connected together by the internal dependency/project structure, the structure itself guides the people to change both projects at the same time. And cross-project structure also prevents form development, as in this trivial cabal.project: packages: ./hnix-store-core/*.cabal ./hnix-store-remote/*.cabal case. IDE simply stopped working because the Core is already done, and Remote work is not, and I can not type into 2 projects at the same time 😄.
   Also monorepo that does cross-project development/structure, as HNix-Store has a testing CI problem - which, Hackage or internal dev versions CI must prioritize & protect? The answer is - both at the same time and which one to look at and to which to close the eyes is situational to the PR, some PRs are to provide support for what is on Hackage, some PRs - are internal development of the master state of projects.

Adding Nix and Nixpkgs on top of it - complicates things much more - now there is:

1. Hackage environment;
2. (future) Stackage environment;
3. Cabal cross-project dev environment (cabal.project);
4. Nix cross-project dev environment;
5. Nixpkgs environment;
6. Local "developer notebook" environment states and variations of the above environments.

• all needed to be satisfied at the same time. And that is impossible - it is not possible or reasonable to support all of them at the same time.
  No wonder 50% of the time is often spent on doing cross-dependency dances.

What CI builds to look too - becomes completely situational.

After I looked at the Store infrastructure - I understood that it is not the way to do monorepo. To keep things robust and effective allows also to keep them simple - the Haskell projects need case 1 - to prioritize the Hackage support and releases, which means to prioritize the frequent stable releases and build and develop things sequentially.

So if HNix would split, and as far I see it would be into 4 packages - the packages should be free from cross-hardcoded of cabal.project or Nix infrastructure. Cabal configs should be independent 100%, above is posted a good example why. Their Nix infrastructures should also be independent as much as possible, maybe import some logic, but do not share the state and not cross-dev-build, cross-dev-build incentivizes the cross-development without releases, complicating the dependency management results in project dev envs brittleness, more effort and time to maintain dependencies in all forms, which wastes time and energy that could been productive, and those complications and requirements of additional effort and dependency management postpones the releases, when there is a Hackage for cross-project interface sharing and testing.

Monorepo means several projects in the same source code version control repo, so it is easier to keep team, dialogs, work, and code of related projects together, everyone shares the same scope and stay informed. Monorepo definition does not involve hardcoding projects together, cross-dev-building them, doing direct cross-changes in master between projects while also trying to support downstream at the same time - that is not simple design nor effective design.

The center of the design and processes between projects should be Either master branch in the monorepo or Hackage.

---

My request is simple, please, lets remove the cabal.project from the root, so at least the Core and Remote Cabal dependencies does not clash during development.

[sorki]

I find it nice that I can just run nix-shell and cabal repl hnix-store-core and work on core or cabal repl hnix-store-remote. While it will only reload one and you need exit and start repl again when changing core and want it to get picked up by cabal repl hnix-store-remote. build tho works seamlessly.

I would say that listing sub-packages in cabal.project is quite common and I've reused this approach in other projects too. Maybe this clashes with newly added hnix-store-{core|remote}/cabal.project? (Tangentially related - is it possible to put cryptohash-sha512 to toplevel cabal.project?)

Could this be fixed on HLS side?

Regarding monorepo approach I like that you can (and have to) change both projects at the same time and don't need to jump thru additional hoops like proposed sequential approach. Due to the nature of these package (or more like one package with sub-packages) I see sequential approach as additional hurdle.

Currently when there are breaking changes to core you have to update and release remote either way. They both can be released separately when there are no breaking changes tho. Ideally we could keep the A.B version the same but I see a problem with that when we need new remote with breaking changes but no changes to core.

I've noticed that currently CI for remote tries to build core from Hackage which I don't really find optimal as that would force us to split PRs that change more packages and do in-between releases.

[Anton-Latukha]

Do not want to start it up on New Years 😄

New Years everybody.

[sorki]

@Anton-Latukha I've added you to mainaters on Hackage for both core and remote. In retrospective, it does make sense to treat both libraries as separate entities. I guess this would work in similar manner when part of monorepo which I don't find appealing currently. Please bear in mind that there are few more hnix-store-* (sub)packages in the pipeline

• currently not very useful on its own hnix-store-db in Database #70 (very useful for testing parsers and builders on big dataset tho).
• possibly -local implementation which needs ^
• split of -remote to separate test harness
• tenative -poly https://gist.github.com/sorki/10a035089e601045789357f37e7ab89c which would abstract functionality provided by -remote/-local and in memory implementations.

Regarding release procedure I would appreciate a short guide (wiki page? RELEASE.md?) with few steps to follow when doing a release, like

• Create tags
• Update changelog
• Publish to Hackage
• Publish on GitHub

maybe including pre-release process.

[Anton-Latukha]

Interesting.

Thank you.

[Anton-Latukha]

With the Setup.hs & hie.yml - projects really were not loading in the most straightforward setup.

To simplify life for myself, I moved from NixOS to classic Linux + ghcup + Cabal + GHC + HLS. It is the most basic canonical setup.

So, when understood that that setup has problems detecting monorepo, and knowing that nix-shell uses cabal development anyway and that Setup.hs gets gradually deprecated by Cabal, so all an ll merged the changes to make builds work at least in the most simple canonical Haskell tooling setup case.

Because if people come around most probably they are using a sort of default workflows: ghcup + Cabal, or Nix-shell + Cabal.

[sorki]

I guess this is resolved as well, closing.

For the last couple of days I'm playing with spacemacs so I see some trouble you probably encountered using it with nix/direnv setups so lots of this makes sense.

[Anton-Latukha]

It is not about the Spacemacs specifics. Never opening reports for specifics.

It is about how the default Haskell tooling works.

Store gives top-level cabal.project.

So if HLS infers / as top-level - it is true, it is indeed was declared by project devs the top-level, since there is cabal.project that cabal v2-build all.

Since HLS processes both projects at the same time - one project breaks the coding in the other.
If during the coding - one of two projects builds break beyond what HLS can stand - HLS stops working for both projects simultaneously.

Top-level cabal.project is &&.

[sorki]

Cool, I'm yet to try HLS.