-
Notifications
You must be signed in to change notification settings - Fork 412
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC] project dependency in dune-project files #1498
Comments
This:
is sufficient for the duniverse to operate -- I can modify the duniverse tool metadata to use this format very quickly. Some thoughts:
A bit ugly, but the user would never have to cut and paste the hash in directly... a tool could generate that when it does the import. |
It may be worth specifying only a top-level target directory and using a (potentially simplified) subset of the URL as a subdirectory name:
which could map to |
What's the point of specifying a vendoring directory instead of letting the duniverse tool choose one? |
Because then dune will unambiguously be able to know which subdirectories are vendored code and which are direct local code. This could, e.g., affect default profiles so that we filter out warnings on vendored code (which is out of the control of the local developer, and annoying to see all the time). It would be good to be precise about the chosen directory layout, as different tools may have different resolution mechanisms for clashing urls. For example, consider |
As I've said in slack, this idea is outside of this feature's scope. Once duniverse creates a lock file for dune, that should be the source of truth for creating a reproducible build. Note that this only a single lock file, it's possible that a project may have multiple lock files (for different platforms for example), and it doesn't make sense to this lock info as project global.
Note that we'd like to add a |
yes, that's fine by me -- I agree with this. |
@andreypopp or @jordwalke could you have a look at this proposal and comment whether that's something you could use for esy? |
I think I see a way that new users could be confused by this proposal. Right now when people use Dune with package managers, they are already confused because there are already two levels of "groupings". First they have to declare "package dependencies", and then inside of their dune configs they create "libraries", and describe the dependencies between these libraries which may or may not span the boundaries of package manager packages. Both of those have utility although it's not obvious to everyone why. I wonder if adding a new kind of dependency "distinct from usual package dependencies as found in package managers" will confuse people more if they already have to wrap their heads around the other two kinds of dependencies. I initially thought your proposal was to just place opam package dependencies in a If generating an opam file from a |
What if those dune projects you depend on did not also include a
The |
The format is not very important, although we are starting to have a multitude of formats and it's becoming hard to follow and maintain, so there is some value is regrouping all the relevant configuration in a single file. Opam files don't have enough information to setup the build of OCaml projects in general. So essentially, we do need to write something like Currently, the grouping in dune works as follow:
With the current proposal, the package grouping becomes less relevant, however it is still convenient when writing integrated tests as it is simpler to declare a dependency on a whole package rather than on individual items. That said, we should indeed allow packages to be declared by other means than by the presence of opam files, as this part is often source of confusion. For the rationale behind this proposal, the idea is that the easiest way to describe an external piece of code is by a URL. Once you consider a project and all the external code it depends on as given by this list of URLs, then you have everything you need to understand the whole project, in particular how to build every binary from scratch. |
I really the idea of using Comments on the proposalI have few comments though.
I think the proposal contradicts these points listed as motivational.
The proposed format doesn't look generic to me. It excludes packages published on opam (and npm). As far as I can see to use this in my project I'd need to make sure I have a clear mapping from currently used packages published on opam/npm to their corresponding source code repositories and more over I'd have to have mapping from released versions to source code refs/branches/commits. I like the simplicity though — one can generate a simple bash script with calls to
Don't existing dune users still want to publish their dune projects on opam? With the proposed
This maybe true if you have every package you use built by dune. But this isn't the case even now when dune is seeing much adoption. Even if OCaml community will adopt dune 100% (I hope this will happen soon!) there will still be software which won't be using (In esy we try to bring such dependencies via esy — this is how it is done with imports vs. dependenciesnitpick: I think using Alternative proposalI'd like to propose an alternative way to specify dependencies in Example:
For each dependency the scheme is:
Where I think it is also useful to allow to specify a list of source/constraint pairs for any specific dependency. Example:
This will allow for different tools to choose what source to use depending on the internal heuristics and/or level of support for different package sources. Note that Dune itself won't need to interpret neither Aside: it might makes sense for dune to understand Example: duniverseGiven the following
Duniverse will filter out unsupported package source for every listed package:
If some package don't have a package source supported by duniverse then duniverse will fail. I think it makes sense for duniverse to allow to locally set/override package source for any of the dependencies. Example: opamThe mechanics is the same as with duniverse. opam would need to be able to read metadata from Example: esySame as above. Discussion: kinds of dependenciesIn esy (and in opam) we have different kinds of dependencies - dev, test, build, doc. It might make sense to include such info in
|
I was wondering about leaving the contents open to interpretation but I hadn't considered allowing several sources. That seems like a good idea to me, especially if opam can use this information as well. Although it's problematic for projects that define several packages, not sure how to deal with these :/ Regarding 100% adoption of Dune, that's indeed entirely up to individual developers whether they want to use Dune or not. All we can do is make Dune as good as possible. However, using Dune systematically does unlock exciting new features such as large scale refactoring. For instance, if all OCaml projects where using Dune, then it would become possible for one person to do a breaking change in the stdlib and then upgrade the entire OCaml universe. It's nice if we can provide this feature at least for self-contained subsets of OCaml projects using Dune.
That would be nice as well, but there is some work to do before this can work. It feels like we almost want to write something like this and let the system sort out what it means: (using github.com/ocsigen/lwt v4.3.3)
(using github.com/facebook/reason v3.3.3) |
@avsm referenced this ticket in a response to me in the Ocaml mailing-list. I have been doing something very similar to this for a few years with my Ocaml build tools and wanted to document my experiences in hopes they could be useful to you. The setup I have is two tools:
An example can be found here:
I think two takeaways that relate to this proposal from my experience:
My use case is different than the one described here of building from scratch. IMO, using opam as the common package definition and fleshing out ideas like |
Version control is the job of git (submodule). I would suggest a scheme for multiple project work together with project dependencies. |
We're working on package management now, and the points of this proposal are acknowledged. For now, to change the resolution of packages, one can configure repositories when solving. |
This ticket proposes the addition of metadata to describe project dependencies in the
dune-project
file. Such metadata will be ignored by dune itself and are expected to be used and possibly maintained by third-party tools such as duniverse and esy. The actual format is open to discussion. If we get something that works for both duniverse and esy we should consider that the format is good enough.The motivations for having this information in the
dune-project
file are:Overview
The goal is to allow users to specify a list of external dune projects to consider in order to perform a build from scratch. By build from scratch we mean in an environment where only the OCaml compiler is available but no other OCaml tools are present. External dependencies coming from third party package managers such as debian, homebrew, opam, npm, ... are not considered by this proposal.
Such project dependencies are distinct from usual package dependencies as found in package managers. Each dune project may define several packages and the dependency graph between such packages may be different from the one between projects. When considering a self-contained workspace, dune should have enough information to synthesise package dependencies, taking this burden away from developers.
Metadata
The metadata consist of a list of external dune projects specified directly as source code repository together with a selector. This will typically be a git url + either a commit hash, a tag name or a branch name. Using plain URLs is a simpler way to identify a piece of external code. In particular, it avoids the need for a central database to resolve abstract names to actual source code.
This list should only include the direct dependencies of the project, not the transitive ones. Transitive dependencies should be discovered by third-party tools such as duniverse or esy by scanning the project dependencies recursively.
Here is how such a list could typically look like:
Recommended usage
When considering the transitive dependencies, it is important that identical URLs have identical selectors. To allow some flexibility it is expected that the selector matches only the major version number. When this selector points to a branch, the tip of this branch should be considered and it is expected that developers only push backward compatible changes to such branches. In particular, while this is allowed, it is not recommended to use
master
as a selector given that themaster
of most project doesn't provide backward compatibility guarantee.When starting a project or adding new project dependencies to an existing project, it might not be obvious to the user how to choose an appropriate set of selectors. They should be able to only write the project URL and let the tool (duniverse/esy) choose an appropriate set for them. Once the tool has made a choice, this choice should be stored in the dune-project file by adding selectors to the project URLs without one.
Support
Dune will provide a
dune.project
library supporting reading and writing theimports
stanza in thedune-project
file.The text was updated successfully, but these errors were encountered: