Support third party ports repositories

[traversaro]

For ports not ready for being included in the main vcpkg repository, or for private ports (use case mentioned in https://github.com/Microsoft/vcpkg/blob/master/docs/FAQ.md#can-i-build-a-private-library-with-this-tool) it would be convenient to have a way to easy specify third parts ports repositories,
as done in homebrew using third party "taps".

cc @j-rivero

[ras0219-msft]

This is absolutely an important feature, and certainly on the roadmap.

Use cases I'm aware of so far:

• Split the main ports into confidence categories (core, extra, community, binary-only). In the future, due to size, we may want these to break into multiple repositories (a la the brew split).
• Team-specific/private port repositories
  • I can see a potential need here to "override" ports in other repositories
• Individual small libraries providing portfiles (something like a PPA: https://launchpad.net/ubuntu/+ppas)

I'd like to find people who can represent each use case we address, to make sure we're actually solving their problem :). It sounds like @j-rivero is hitting the third point.

My current best solution idea is to require one level of subdirectory under ports\. We'd accomplish point 1 above by moving all the existing ports into subfolders accordingly. You could easily add additional port repositories hosted in git (or otherwise) by cloning them into this folder.

An important point that I believe affects all solutions is to make the organization of these folders entirely user-facing and completely transparent to the portfiles themselves. There needs to be a resolution process to consolidate core/boost and my-company/boost, but portfiles cannot depend on core/boost -- they only see boost.

[adam4813]

For point 2 team building against certain versions of libraries, such as game development teams. By providing a link to their ports repo contributors can use a certain version or set of libraries without having to worry about breaking builds.

[adam4813]

Is this something to move forward with because I can spec out a solution for this if so.

[ras0219-msft]

Yes, we should move forward on this. I believe there is some overlap with #164, especially around the need to "override" an existing port.

I've discussed this with @alexkaratarakis offline and we did come up with a few solutions which address both #164 and this, though we were waiting for a bit more discussion/scenario gathering on the other issue before moving to the solutions stage.

The best solution we have so far is first to have the above ability to nest directories underneath ports\. Ports must still be uniquely named, the directories do not act as namespaces. Then, there is a per-vcpkg-enlistment "aliases" file which enables you to remap how dependencies are resolved. This would allow redirections such as openssl -> boringssl (enabling #164) or boost -> boost-mycompanyfork enabling the per-team overrides mentioned above. This also would enable a nicer "pin to older version" experience, such as boost -> boost-1.55. Note that the aliases apply only to build time dependencies. Binary packages are forever stamped with the true dependency so we don't use the wrong binaries.

Alternatively, directories do act as namespace for the aliases file and conflicts always resolve to the "core" directory. I don't think this would give a very good experience for teams however, since an update to core could suddenly redirect one of your private packages to the "core" package.

What solution did you have in mind?

[adam4813]

I truly believe that this issue is aiming at allowing me to specify check at this location for a list of portfiles to use instead, and not so much sit side-by-side with an existing portfile. But as for you second option this could be more natural and intuitive at the graphical level allowing someone to visually see which alternatives are installed. I think that having a "core" version that is a fallback is exactly what we don't need. By forcing the using of specific versions there won't be any surprises like you are suggesting. So when the user installs a package there's a configuration file created that says which version they are using and then an upgrade would still pin them to that version until they issue an upgrade command.

[ras0219-msft]

We do currently have a --vcpkg-dir parameter which allows you to redirect everything except the executable (ports, downloads, installed, triplets, etc). This could be made more granular, allowing individual redirection of the ports directory. My intuition says that redirecting the ports directory without redirecting the installed directory is not such a great idea, since you either lose the connection of where the package came from (if you don't have the config file mentioned) or you lose the relocatability of the directories (since you'll encoding paths in a file).

What's the story you're thinking of when you speak of the configuration file specifying versions? What happens when I run install, upgrade, remove, install with various parameters?

[adam4813]

You would have to be explicit in your commands such as install boost-1.62.0. To better facilitate this the list command would need a way to filter and there would need to be extra lines in the control file to specify the then current version so a person could just install boost to get boost-1.62.0 however any command after that e.g. remove install upgrade would need the explicit version in it's name.

[ras0219-msft]

To make the conversation a bit more understandable for me, I'd like to use "version" to describe older and newer versions of the same portfile and use "alternatives" to describe other portfiles which are acceptable substitutes.

So, with what you are proposing, my experience would be something like this?

> vcpkg search
openssl
libressl
boringssl
> vcpkg install openssl
-- Note: choosing libressl for openssl due to alias file --
Installing libressl....
> vcpkg list
libressl
zlib
> vcpkg remove openssl
Openssl is not installed
> vcpkg list
libressl
zlib
> vcpkg remove libressl
Libressl is removed
> vcpkg list
zlib

The only change i would make is that I think all the user interactions should be redirected along the aliases/configuration file choices. So vcpkg remove openssl in the above transcript would remove libressl transparently.

Back to your initial situation though:

specify check at this location for a list of portfiles to use instead, and not so much sit side-by-side with an existing portfile.

Does the --vcpkg-dir option handle that sufficiently, since you are looking to completely replace the set of portfiles? The way we imagine more controlled environments would use this is by either using git subtree or git submodule to own and manage a copy of the ports tree and triplets that they are building and shipping against.

[Orvid]

Alright, so my use-case would be something like this (names subject to change):

vcpkg add-ports facebook/fb

Just as brew tab facebook/fb looks for https://github.com/facebook/homebrew-fb.git, this should look for https://github.com/facebook/vcpkg-fb.git and clone it locally.

The structure of https://github.com/facebook/vcpkg-fb.git would be something like this:

Readme.md
boost/CONTROL
boost/portfile.cmake
boost/user-config.jam
libevent/CONTROL
libevent/portfile.cmake
pthread/CMakeLists.txt
pthread/CONTROL
pthread/portfile.cmake

You could then do:

vcpkg install facebook/boost:x64-windows facebook/libevent:x64-windows facebook/pthread:x64-windows

To install the 3 ports in the repo. If the boost port from the main vcpkg repo, or from a different port repo, is installed, an error should be produced to say that they conflict with each other and that the currently installed port must be uninstalled before facebook/boost:x64-windows can be installed.

For the sake of scaling, brew allows you to install ports without explicitly specifying the repo to install from if no other port with that name is provided in the main repo or any of the other port repos. It might be reasonable to allow the same thing with vcpkg, which would make the previous command do the same thing as this, as long as nothing else is providing a port named libevent or pthread:

vcpkg install facebook/boost:x64-windows libevent:x64-windows pthread:x64-windows

Implementing this would allow me to publish the ports that I currently have, as well as (hopefully) the changes to existing ports, which I can't currently do as our legal team has some concerns about the CLA, and also do not want us creating public forks.

[donny-dont]

Any progress on this? If so I'm interested in using this for some third party dependencies when building WebKit on Windows

[ras0219-msft]

The immediately available way to handle this (publishing ports without upstreaming them to the wider community here) is to host a ports\ directory that users can copy over their local ports\.

For example:

• https://github.com/CdecPGL/Additional-Packages-for-Vcpkg
• https://github.com/clvn/vcpkg_extras

While there isn't a first-class command (vcpkg tap xyz), it is still pretty simple as:

> git clone <url>
> cp -r donny-ports/ports/* /path/to/vcpkg/ports

[donny-dont]

@ras0219-msft thanks for the examples! I'll have to give that a go for the time being.

[martin-s]

What I have in mind would be something like this:

• ./downloads
• ./installed
• ./ports
  -- zlib
• ./triplets
• .vcpkg-root
• vcpkg.exe
• 3rdparty
  -- ./installed
  -- ./ports
  --- tiff
  -- .vcpkg-root

So, I have one vcpkg.exe and several port folders. zlib port is build on the root folder. tiff port is build on subfolder "3rdparty", but can resolve dependency (e.g. to zlib) to root folder (maybe by using .vcpkg-root, write the root folder in there in order to link to that). Think of a vcpkg that can take build installation of another vcpkg build folder. Downloads folder should be used only once in case an other vcpkg build folder is referenced in the .vcpkg-root file in 3rdparty.
Is the idea clear?

Pseudo commands would look like:

vcpkg.exe install zlib
...build zlib
vcpkg.exe --vcpkg-dir ./3rdparty install tiff
...zlib installed
...build tiff

[MichaelMitchellM]

Is it possible we could get a system similar to apt, pacman, and yum, where one could add a repository to their local vcpkg?

A repository would be any git repo accessible from the user's local vcpkg that has a top-level ports directory containing vcpkg packages.

Adding the repo to the local vcpkg would be something like vcpkg add-repo <repo-url> [repo-credentials]

My use case would be managing private internal packages.
It is possible that a shared filer could be set up that contains our private ports, but that is not good infrastructure. It would be preferred if we could set up a private git repo and point our vcpkg at that.

[vicroms]

The new --overlay-ports option should address some of the discussion points in this issue (implemented in PR #6981).

Example:

./vcpkg install zlib --overlay-ports=C:\ports\3rd-party

See specification for details of how to use this feature.

[j-rivero]

The new --overlay-ports option should address some of the discussion points in this issue (implemented in PR #6981).

Ouuh, thanks @vicroms . Good news.