[vcpkg] RFC: Manifests

[strega-nil]

We are currently working on a new feature: manifests. They are similar to manifests from other languages: Cargo.toml, package.json, etc. This specification lays out how we've thought about designing the manifest so far; please add your comments so that we might have the best experience possible :)

A note: there are two major features that pretty much depend on manifests, but which will be defined in separate RFCs: versioning, and federation. This manifest is designed such that those features are possible to fit on top of it, but it does not attempt to define those features right now.

Currently, this feature is being developed at strega-nil/vcpkg/manifests; please come check it out for very very pre-alpha demo-friendly support of this! An example of this feature in use is at strega-nil/vcpkg-example 😄

rendered

[edhebi]

It looks cool! I don't have any constructive feedback regarding the manifest itself, but the nested markdown lists are broken in the Definitions section.

I can submit a fix if you want. I suppose I should PR your own branch for that ?

[strega-nil]

yeah, that works! sorry I can't work on it right now, I'm doing work on macOS pre-10.15 support -.-

[petersteneteg]

Cool! so looking forward to this

One question the vcpkg_modules folder seems to go in which I assume would usually be the root directory of the project itself. To make sure the manifest will be version controlled. But when using CMake the best practices now days seems to be to keep the project folder clean of any build product, ie. out of source builds. For the example in presented one would usually have:

projdir/
  build/
  example/
    src/
      main.cxx
    CMakeLists.txt
    vcpkg.json

or something like that. I would prefer if the vcpkg_modules folder was either in the build folder or at the same level as the build folder. At least I would never want it to be inside of example.

[strega-nil]

@petersteneteg I simply followed existing practice for Javascript; however, there's a good point there that, for compiled languages, it doesn't make sense to commit your vcpkg_modules directory. However, it's unclear where any specific build system (or even user! it's user-chosen with build systems like cmake) will place their build trees, and thus vcpkg install wouldn't be able to put its vcpkg_modules folder underneath the build tree. Thus, I'm not sure of how to do this otherwise...

Additionally, I don't think that out-of-source-tree builds are generally accepted practice with modern CMake. That seems to be a leave-over from autoconf, maybe? But vcpkg and ninja-build, as two decently modern examples, both expect a person to build in-source-tree.

[traversaro]

Thanks a lot for working on this! I have added a few comments inline, I hope they can be useful.

[traversaro]

Additionally, I don't think that out-of-source-tree builds are generally accepted practice with modern CMake.

If out-of-source-tree builds means "keeping the build directory not inside the source directory", I don't think it is uncommon. Probably it is just my bubble, but out-of-source-tree builds are standard practice when using build_tools such as colcon or CMake-based superbuilds. Furthermore, it is what you get by default when consuming dependencies via CMake's FetchContent, and if i am not wrong vcpkg itself builds its own ports out-of-source-tree when using vcpkg_configure_cmake.

[strega-nil]

@traversaro sorry, I meant the other way around -- it's not that out-of-source-tree builds aren't accepted practice, but that in-source-tree builds are totally valid.

Also, for multi-level super-projects, what I might expect is something like Rust's workspaces: each sub-project has their own vcpkg.json file, with the superproject defining a vcpkg.json file that depends on each of the sub-projects; then, a vcpkg install will install the dependencies into the root. However, this seems likely to require a way to define sub-projects, since you don't want to think that the sub-project's folder is where one should install dependencies... interesting... We should probably look towards Rust's workspaces for ideas on how to solve this. However, I don't think this is necessary to solve for this initial MVP.

[kobalicek]

Feedback:

• You need revisions - since package information can be maintained out of project there has to be some information about the revision. Imagine you fix vcpkg.json, but not the main project, the revision of the package should change, but it's still for the same version of the library.. Conan has this, for example.
• Out of source build should be the only build possible. I'm not even sure CMake still allows in-source builds - nobody really wants in-source builds these days anyway, it's a bad practice.
• Each directory one vcpkg? This is backwards - a nightmare everybody faced when having large cmake projects, for example, where you have 100 CMakeLists.txt files - find the right one in an editor, find the right one in 20 open tabs...
• I think it would be nice if vcpkg was unobtrusive - I don't like it couples with CMakeLists - it gives users no choice, which is something that I won't accept, but I may still provide vcpkg.json in my project for people that want to use it. So it would be cool if it worked like other package managers - you install dependencies, then you build the project, but you won't do both at the same time.

[strega-nil]

@kobalicek

• port-version is the way to do this
• I disagree strongly; it is very much not bad practice, and it's literally the recommended way to build CMake. I have no idea where people got this idea from see edit
• You don't have to have multiple vcpkg.jsons in a project; in fact this initial specification does not support multiple vcpkg.jsons in a project. It's expected you'll have one vcpkg.json per-project.
• It doesn't have to couple with CMakeLists.txt, that's just one possible usecase. I should really add an example of this! edit: done :)

Edit: okay, someone explained to me what "in-source builds" are. Sorry for being confusing! I'm not trying to argue that in-source builds are good, but that in-project builds are good. I hadn't even considered in-source builds to be a thing that one might do... the fun of being young, I guess!

[petersteneteg]

Edit: okay, someone explained to me what "in-source builds" are. Sorry for being confusing! I'm not trying to argue that in-source builds are good, but that in-project builds are good. I hadn't even considered in-source builds to be a thing that one might do... the fun of being young, I guess!

But why put the vcpkg_modules folder "in-source" then? the vcpkg_acquire_dependencies() should know about the build directory...

[strega-nil]

@petersteneteg The vcpkg_modules is not in-source, just in-project. Quite different :)

[petersteneteg]

@petersteneteg The vcpkg_modules is not in-source, just in-project. Quite different :)

Citing Professional CMake: A Practical Guide https://crascit.com/professional-cmake/
Chapter 2.2

This book will always use out-of-source builds and will follow the pattern of the source and build directories being under a common parent. The build directory will be called build, or some variation thereof.

A variation on this used by some developers is to make the build directory a subdirectory of the source directory. This offers most of the advantages of an out-of-source build, but it does still carry with it some of the disadvantages of an in-source arrangement. Unless there is a good reason to structure things that way, keeping the build directory completely outside of the source tree instead is recommended.

Hence I would strongly prefer to keep the vcpkg_modules folder completely outside of the source directory. And by source directory I mean the root of my repository and the same directory as were the manifest file would be in.

[kobalicek]

I think they should look into Conan - not saying Conan is the best tool, but it solves a lot of problems and maintains its own cache that is configurable through $CONAN_USER_HOME - so by having different env-vars you can pretty much isolate it. It has much more additional things that are very usable - like propagating environment variables, library directories, binaries, etc... through projects.

[strega-nil]

@petersteneteg That's not what that quote is saying, I don't think: the "source directory" is not the directory which the project is in. The source directory is the directory containing the literal source files -- .c, .cxx, etc. -- usually called src or source. The build directory and the source directory are distinct, and are in the same directory as any manifests or main build files (CMakeLists.txt, for example). I can't speak to that specific book, as I don't really want to buy it and I don't personally have access to it, but I think it's important to realize that, for example:

• vcpkg recommends builds at the same level as CMakeLists.txt
• ninja-build recommends builds at the same level as CMakeLists.txt
• CMake recommends builds at the same level as CMakeLists.txt
• LLVM recommends builds at the same level as CMakeLists.txt

so it's extremely common practice to have a project tree that looks like:

src/
  <source>...
build/
  <build>...
README.md
CMakeLists.txt

On the other hand, it does make sense to have the vcpkg_modules directory under the build/ directory, so it's likely that the CMake integration will do that.

[petersteneteg]

CMake defines the "Source tree" as the directory with the top level CMakeLists.txt. No other parent directory would really matter for CMake. https://cmake.org/cmake/help/v3.17/manual/cmake.1.html#manual:cmake(1)
For the "Build Tree" is says:

To maintain a pristine source tree, perform an out-of-source build by using a separate dedicated build tree. An in-source build in which the build tree is placed in the same directory as the source tree is also supported, but discouraged.

Here I interpret separate as not a subfolder.

The Book does the same interpretation.

Using a subfolder might be common practice. But that does not make it good practice.
And it would be unfortunate if vcpkg enforced that pattern.

[kobalicek]

I don't really see a difference between:

src/
  <source>...
build/
  <build>...
README.md
CMakeLists.txt

and

src/
  CMakeLists.txt
  <source>...
build/
  <build>...
README.md

Both are basically out-of-source builds, because the build tree doesn't intermix with source-tree. Every project will decide where the CMakeLists.txt will be, so vcpkg shouldn't care anyway.

[meastp]

Thank you for working on this!

Am I correct in assuming that this could also be used even if one only uses vcpkg to manage/acquire third-party dependencies?

For example, we develop in Visual Studio and use MSBuild to build our project(s)...

Our use case:

myprojects/ //project root with multiple projects
  proj1/
  proj2/
  proj3/
  x64/Release/ // custom build output folder for all .exe .dll and third-party binaries
  myprojects.sln
  vpkg.json // all dependencies for all projects

I'm guessing that this works with vcpkg integrate install, but how are different versions of packages handled and how will this work with binarycaching #11204?

Ideally I'd like to create a vcpkg.json file that is used during CI and on developer machines to acquire third-party dependencies and use binarycaching to ensure that they all use the same version and avoid building from scratch every time.

[strega-nil]

@meastp yep, absolutely! before this is stabilized, we will support every build system we support currently 😁

[meastp]

@meastp yep, absolutely! before this is stabilized, we will support every build system we support currently 😁

That's great!

[david-antiteum]

It seems that vcpkg.json will have a dual role, when used on a port and when used by a developer to list dependencies of an application. In the second case, the name and version are not required or available elsewhere (for example, in the project entry CMakeLists.txt).

[strega-nil]

@david-antiteum it's important to note that it's not actually a dual role -- it's just expanding what a "port" looks like.

[meastp]

@strega-nil We have custom ports that are not present in the official vcpkg repository. Could one point to where (github url, file share etc.) the port "definition" is?

For example, we have both open source ports and binary proprietary ports... :)

EDIT: This is actually part of the feature description, but I can't see it specified in this PR?

Vcpkg will support a manifest file that can specify all your dependencies declaratively and can be checked into source control. The idea is straightforward: you specify your dependencies, versions, and where they can be acquired from (in the case of private dependencies) in a file called vcpkg.json. That file can exist as part of your source code repo, and vcpkg can read the file to immediately acquire everything you need in a consistent way, every time.

I see <dependency.git> in the specification, but there needs to be a way to point to git urls for top-level dependencies as well (for example myproject will have a vcpkg.json-file in the root folder, which contains binarylib as a dependency and ocilib as a dependency - neither are present i the vcpkg repository.

[strega-nil]

@meastp for this initial RFC, we aren't going to add that feature; however, it's planned for very soon afterwards :) We call it "federation" in the wiki. I just didn't want to try and overload this so nobody gets manifests while we get federation specified and working.

[meastp]

@strega-nil Ah, ok, that's fair :)

I currently use a fork of vcpkg where I have added the ports missing from official vcpkg (binary and open source) that we need. What I think you're saying is that:

• before the package federation feature is done, I'll need to keep using my fork and initiating vcpkg.exe (to build dependencies) from my fork (that I have downloaded/cloned and built locally) when working with vcpkg.json in myprojects (example above)

• when the package federation feature is done, I won't be dependent on managing my fork at all as long as I can point to the correct path for each dependency (url, path etc) in vcpkg.json in myprojects.

this ties in nicely with vcpkg.exe being included in Visual Studio - then I can use that even though I use dependencies that are not part of the official vcpkg repo.

Are my assumptions correct? :)

[strega-nil]

Your assumptions are correct! Although, if you want to avoid maintaining a fork, you may want to look into --overlay-ports until such point as we add federation to the manifest.

[ghost]

Thanks for working on this!

If I understand correctly, the vcpkg_modules folder will be created for each cmake project and contain the binaries of it’s versioned dependencies. Will there be a way to first check whether a version is already installed “globally” by VCPKG to avoid having the same copy of the library on each dependent project?

[strega-nil]

@daniel-piro that's not a feature I'm interested in adding personally; I think that having the installed directory be completely different will allow one to separate the two really cleanly. For what it's worth, we will support binary caching, which means if you already have it installed anywhere, you'll be able to cache binaries, so that means it's just an unzip, not a compile.

[strega-nil]

I'm going to Final Comment Period this! This manifest will be merged on Monday, June 1 (a week) if no one else has any comments :)

Please add any other comments you wish to! Thanks all!

[nightlark]

This is great! It looks like manifests and federation will help alleviate the biggest issue we've run into, which is challenges submitting a PR to the vcpkg repository for ports that we want other people to be able to easily use in their projects.

[seanyen]

@strega-nil Just curious, for the vcpkg.json manifest, does it support (or plan to support) the following workflow?

1. create an empty folder as the root for a new project.
2. create a vcpkg.json under it.
3. do vcpkg install under the folder and magically vcpkg will build and produce the installed for this new project?

I am viewing this manifest ting from my experience in npm, pip and conda, and hope the vcpkg workflow won't be deviated too much from those popular package managers too. And btw, this is a great work and happy to see vcpkg is moving toward this direction.

[strega-nil]

@seanyen this is absolutely a valid thing to do.