Find a testing solution

[Telokis]

As stated in #41, we should find a proper way to test the portfiles.
An ideal system would be to automatically test new/updated portfiles after each push and ignore those which were untouched.

[KindDragon]

Maybe at least do dry-run for all packages? To check syntax error in portfile.cmake

[traversaro]

As highlighted in #77 , a possible bug/regression is that a given package compiles and install correctly, but can't be found from a CMake downstream project.

Having per-package simple tests (like the example attached to #77) that are compiled every time the package changes could help preventing such regressions.

[KindDragon]

conan, for example, has test_package command that which is trying to compile a simple project that uses the package

[Telokis]

Our main issue, there, is to handle scalability.
As the number of provided libraries grows, the time taken to pass the testing will get bigger and bigger.
We need to find a way to only test the packages that have changed.

[ras0219-msft]

@KindDragon @traversaro I really like the idea of checking in a simple project which should work if the package was built correctly. Perhaps a ports\<port>\test\ directory which can contain a CMakeLists.txt that builds an executable called main.exe? In contrast, it would be nice to avoid magic here wherever possible.

One issue that would need working around is that we currently only support pulling libraries from the installed\ directory, but we'd really rather check the packages before installing them. Off the top of my head, this should be as easy as tacking another directory onto the CMAKE_PREFIX_PATH variable, but it'll require some trial and error.

@Telokis Perhaps it could be as easy as just running git diff HEAD~ --stat and grepping the output:

PS D:\src\vcpkg> git diff HEAD~ --stat
 toolsrc/src/commands_help.cpp | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

If we look for changes of the form ports/<name>/....., that would list every port that was touched. This still would have scaling problems w.r.t. deep dependency chains, and lots of packages will be testing boost, but it should help things quite a bit. If we use AppVeyor like your PR suggested, we can also (possibly) cache a copy of the system.

[Telokis]

@ras0219-msft Yeah, you're right, parsing the output of a git command might be enough.
This means that we should test each push properly. Otherwise, we might miss a portfile modification.

[MikeGitb]

I'd like to point out, that just testing changed port files - while certainly the most important part - may not be enough. You also need to check if the dependent ports still build when you e.g. upgrade a library.

Also, as I mentioned in the referenced issue: Couldn't Microsoft perform a regularly scheduled build of all libraries on their internal servers to check for compatibility issues.
This wouldn't deliver the same kind of feedback as a per port test that would run for each PR, but it complement them (think about Unit vs integration testing)

[ras0219-msft]

From #923:

As there is no visible ci going on I was wondering if the MS Team actually checks if all the portfiles work correctly (on which platform / with which compiler / are you running any tests / any test projects?). And if the library versions in the master branch play "nice" together or if there are any incompatibilities.
Especially when less experienced people like me are trying to write a non-trivial portfile or upgrading an existing one it would be nice to have more feedback than just: "It seems to work on my machine"

CI / unit testing / integration testing

While nowhere near as visible as it needs to be, we do have a limited amount of CI that runs on every change to the master branch. You can see some checkmarks beside many of the commits in https://github.com/Microsoft/vcpkg/commits/master. I've also just added the current status badge to our front-page README as well.

This CI system does the following (Win10+VS2017):

• bootstrap
• recursively purges all out-of-date libraries
• build a "core" set, including their dependencies under several flavors
  • This core set includes azure-storage-cpp opencv protobuf chakracore libuv and several others

[todo: we should check in the script for this, at least as documentation.]

We also do (irregularly) build all ports -- the results from last time showed 6 failing ports (and 3 cascaded) out of ~170 under x86-windows[1]. Most issues we see day-to-day are unfortunately ones that this style of CI system can't reasonably catch: they're due to unique combinations of client and port, such as having files under "C:\OpenSSL" (which confused find_package(OpenSSL) -- fixed!) or having a "site-config.jam" (which confused the boost build -- not fixed ☹️).

It would be really awesome to have a dashboard for this! Something like a "sticky" GitHub issue would be ideal, but unfortunately the closest we have to that is perhaps a wiki page? We need to do some research to find the best option. I'd really appreciate links to all your favorite projects in this regard -- who does this the best?

Testing techniques

Today we don't run any tests beyond the build+post-package lint for libraries built inside Vcpkg. This is definitely an area which needs improvement! There are a few obvious ways we can improve testing:

• Use the tests inside the library -- what to do about flaky tests?
• Provide an example program for each port -- would serve as great documentation!
• Build all the ports that depend on the "port under test" -- the CI system above does this for core libraries like boost and openssl, but it would be good to expand this for all update PRs
• Expand our automatic post-package tests -- these are great for catching packaging errors like "I accidentally built the wrong architecture" or "I forgot to copy over the include files"

As mentioned in a previous comment, there are some additional core capabilities we need to support some of these. We would need to expand the buildsystem integration to allow pulling from a package's temporary directory before installation.

[1] The failing ports were:
clockutils (since fixed)
hdf5 (since fixed)
libmysql (since fixed)
nana (since fixed)
qt5 (since fixed)
spirv-tools

[traversaro]

@ras0219-msft perhaps it is intended to be like that, but if you click on the checkmarks in https://github.com/Microsoft/vcpkg/commits/master , you need to be logged (I guess with some Microsoft/Visual Studio account) to see the results of the CI.

[jumpinjackie]

Just to add my 2c

I've been maintaining a repo (https://github.com/jumpinjackie/vcpkg-port-tests) of little test applications that are sanity tests for vcpkg ports that I am personally interested in or port additions/updates from me that are currently in the PR queue.

Maybe you could have something like my test.ps1, but parameterized so that you can run it against any incoming PR and the ports that they affect.

[cmpute]

In terms of #3431 , I think port testing should be considered as default operation and a portfile has to either run some tests (using vcpkg_test_cmake or other scripts) or define VCPKG_POLICY_NO_TEST just like other policies. And afterwards, this could be confirmed by post-build validation of vcpkg.

[miurahr]

@cmpute It is better to provide not only vcpkg_test_cmake but vcpkg_test_msbuild or other scripts for test coverage on varying build environment. An idea to define VCPKG_POLICY_NO_TEST is good and I'd like to ask a way to define a policy such as "some package is provided for msbuild but no guarantee for cmake build."
It is time consuming and complex task for contributors to test it on every different platform and build system matrix, so contributors may test only on their favorite platform and VC version with favorite build system.

[MathiasMagnus]

A seemingly dumb question: with Azure Pipelines having unlimited CPU time for building OSS projects for CI/CD purposes, can't we hook all of this testing into that?

#921 should also be added to testing. The route proposed by #3107 is a nice addition for testing the validity of a build, but it doesn't indicate that the install layout is compatible with the potentially patched package config files. That only becomes apparent once an external application tries to consume it in the intended way, which brings us back to providing a fleshed out minimal example of invoking a single function from the dependency, proving headers and linking is right.

And yes, it should be done for all downstreams as well. Building all of Vcpkg in this manner centrally is not feasible IMHO. It can only be done incrementally, enforcing it after a given point for all package updates.

My problem is that nobody is bugging me to fix (or even find out if it needs fixing) my ports for Linux. I feel that a significant proprpotion of the packages have not been touched since Vcpkg went cross-platform and many packages have not been tested on Linux, not to mention OSX. CI/CD could really help out in that regard.

[strega-nil]

Closed as complete.