Struggling with differences in abi_info and build failures with binarycaching

[meastp]

I am trying to use vcpkg for building in Azure Pipelines (CI, Microsoft Hosted Agent windows-2019 and vs2017-win2016) and to be able to reuse the binary packages produced on my dev machine.

Everything except two packages compile successfully in Azure, but the binary package hashes are different. The two packages that fail (libpq and ocilib, with an error about corrupt patch) does compile successfully on my dev machine.

Here is an example from the abi_info on ocilib:

Dev machine (builds ok):

CONTROL 9241a03e2bb576c99b871f63a5b0c8b2ed4c2a95
cmake 3.19.2
features core
portfile.cmake 0657a4f80fd0e53da0ec80d0aa805ad7f53211ae
post_build_checks 2
powershell 7.1.1
triplet x64-windows-v141
triplet_abi 3e1e1ba8308fbe92aa7777d17951828b7098c537-a1c0eabb0c5177b6a8fb97a58ae398880c47b352-b9e2480dca17932aec14cc7258376a7b281aa769
vcpkg_configure_make e1a003f885d4091e1e3fab0f82be37ab3d8d88a7
vcpkg_fail_port_install 69f61f498784f900b570c22b1e8bee2ca8995840
vcpkg_from_git 99e83c6ffac2dd07afaf73d795988ca5dec2dc8b
vcpkg_from_github a9dd1453b8873c9702731c6c36e79fb0ab6ae486
vcpkg_install_make 6b7345be4d3363747f6d75839e87d441025f04d5
vcpkg_install_msbuild b1f007c6cc36e1fb840cba9ee18057b3a8d648e6

CI (build fails):

CONTROL 9241a03e2bb576c99b871f63a5b0c8b2ed4c2a95
cmake 3.19.5
features core
portfile.cmake 0657a4f80fd0e53da0ec80d0aa805ad7f53211ae
post_build_checks 2
powershell 7.1.2
triplet x64-windows-v141
triplet_abi ed6353d74e9e730073fb23380c6237636df93252-a1c0eabb0c5177b6a8fb97a58ae398880c47b352-b9e2480dca17932aec14cc7258376a7b281aa769
vcpkg_configure_make e1a003f885d4091e1e3fab0f82be37ab3d8d88a7
vcpkg_fail_port_install 69f61f498784f900b570c22b1e8bee2ca8995840
vcpkg_from_git 99e83c6ffac2dd07afaf73d795988ca5dec2dc8b
vcpkg_from_github a9dd1453b8873c9702731c6c36e79fb0ab6ae486
vcpkg_install_make 6b7345be4d3363747f6d75839e87d441025f04d5
vcpkg_install_msbuild b1f007c6cc36e1fb840cba9ee18057b3a8d648e6

I see that these are different:

• cmake patch version
• powershell patch version
• the first part of triplet_abi

I'm hoping that patch versions don't affect the binary hash version of the binary packages?

Could you explain the last difference (triplet_abi)? I would like to produce binary packages that I can reuse between machines, and (obviously) successfully build libpq and ocilib in Azure.

[meastp]

btw, this is the message i get for building libpq (both v141 and v142):

Starting package 157/187: libpq:x64-windows-v142
         Building package libpq[core,openssl,zlib]:x64-windows-v142...
         -- [OVERLAY] Loading triplet configuration from: D:\a\1\s\triplets\x64-windows-v142.cmake
         -- Installing port from location: C:\Users\VssAdministrator\AppData\Local\vcpkg\registries\git-trees\d68450074a2fbc7a2aaeb23b9f4fdda0532e5897
         -- Downloading https://ftp.postgresql.org/pub/source/v12.2/postgresql-12.2.tar.bz2 -> postgresql-12.2.tar.bz2...
         -- Extracting source C:/vcpkg-latest/downloads/postgresql-12.2.tar.bz2
         -- Applying patch patches/windows/install.patch
         -- Applying patch patches/windows/win_bison_flex.patch
         CMake Error at scripts/cmake/z_vcpkg_apply_patches.cmake:57 (message):
##[error]Applying patch failed(0,0): Error : corrupt patch at line 39
     5>Applying patch failed : error : corrupt patch at line 39 [D:\a\1\s\src\myproj.vcxproj]
         
         Call Stack (most recent call first):
           scripts/cmake/vcpkg_extract_source_archive_ex.cmake:144 (z_vcpkg_apply_patches)
           C:/Users/VssAdministrator/AppData/Local/vcpkg/registries/git-trees/d68450074a2fbc7a2aaeb23b9f4fdda0532e5897/portfile.cmake:65 (vcpkg_extract_source_archive_ex)
           scripts/ports.cmake:139 (include)

the message for ocilib is more or less equal. I can't see how the patch is corrupt. (And, as I mentioned, this only happens in the microsoft hosted azure pipeline)

[PhoebeHui]

@meastp, do you install git locally, if so, what's revision do you use?

It may be relate to the EOL, I noticed that EOL of the failed patch is CRLF, and the previous passed patch are LF.

[meastp]

@PhoebeHui

locally: 99dc49d

In Azure I have both tried using the provided revision, and git clone https://github.com/microsoft/vcpkg.git vcpkg-latest. Both give errors.

Yes, it did look like it might be EOL related when I looked closely at the patches. How to resolve this?

[PhoebeHui]

@meastp, could you manually update the EOL of 'ports/libpq/patches/windows/openssl_exe_path.patch' file to LF, and see if the patch could apply successfully?

[meastp]

@PhoebeHui I opened all windows patch files in notepad++ (which shows line endings), and when comparing the patches with the source files they apply to, I see differences in line endings in multiple patch files. Not sure if this is a problem for all the files:

• openssl_exe_path.patch (as you noticed, the patch is consistently crlf, the source is consistently lf)
• msgfmt.patch (the Install.pm file contains a mix of line endings, and is not consistent)
• python_lib.patch (the patch is consistently crlf, the source is consistently lf)
• Solution.patch (the patch is consistently crlf, the source is consistently lf)
• Mkvcbuild-static-lib.patch (the patch is consistently crlf, the source is consistently lf)

[PhoebeHui]

@meastp, I can't repro this issue locally, so could you help confirm if these patches could apply successfully when update all EOL from CRLF to LF in your machine?

[meastp]

@PhoebeHui I could never reproduce this locally - it only happens on the Microsoft hosted azure pipeline agent. I'm not sure why...

[meastp]

@PhoebeHui could this be a git (global) setting on the agent that handles crlf/lf differently than on our local machines?

[PhoebeHui]

It may be relate to the git gloable setting, @strega-nil, do you know more about this?

Relate to https://github.com/microsoft/vcpkg/blob/master/scripts/cmake/z_vcpkg_apply_patches.cmake#L48

[meastp]

I have added the following git config to my pipeline yaml script:
before:

    - powershell: |
        git clone https://github.com/microsoft/vcpkg.git vcpkg-latest
      workingDirectory: 'C:\'
      

after:

    - powershell: |
        git config --global core.autocrlf false
        git clone https://github.com/microsoft/vcpkg.git vcpkg-latest
      workingDirectory: 'C:\'

We will se if this makes a difference

[meastp]

@PhoebeHui The core.autocrlf false did work for libpq. I will file a separate issue for ocilib, which is unrelated. Thank you for helping me with this! :)

Could you please explain the differences in the first part of triplet_abi so I can perhaps fix it/consolidate and reuse the binary packages between CI and my dev machine? :)

[PhoebeHui]

From https://github.com/microsoft/vcpkg-tool/blob/main/src/vcpkg/build.cpp#L444, the differences in the first part of triplet_abi seems due to the triplet_entry.hash is different.

@ras0219-msft, could you please take a look at the abi info?

[meastp]

@ras0219-msft perhaps the parts making up (at least the first part of) triplet_abi could be printed as text as well (like the cmake version and powershell version is)? I'm really interested in keeping the same binary package between CI and dev machine(s) (and making sure all have identical abi_triplets when possible).

[ras0219-msft]

We do have a .gitattributes which should have ensured all files are checked out using the original line endings, so I'm very confused as to why #16615 (comment) made a difference. Assuming that's the problem, it's likely that the triplet file you're using has different line endings across your machines, which makes the hash different.

I also agree it'd be good to improve visibility of the precise factors; maybe improving https://github.com/microsoft/vcpkg/blob/master/docs/users/binarycaching.md#implementation-notes-internal-details-subject-to-change-without-notice is the right first step.

[meastp]

@ras0219-msft Thank you for the explanation. So, you're saying that the first hash of the triplet_abi is computed from only the triplet file? Or could this be caused by multiple reasons? Would the difference in CMake versions 3.19.5 vs 3.19.2 contribute at all?

I would think being able to reuse the binary packages across machines is pretty important, so more documentation would help troubleshooting/pinpoint the issue.

[aee11]

I also encountered the error : corrupt patch at line 39 on the libpq win_bison_flex.patch locally. That file has LF line endings under ports/libpq but under the versioning directory C:/vcpkg/buildtrees/versioning/versions/libpq/04f39fd5eb9744d0d5a649098fe92281028d30f1/patches/windows/win_bison_flex.patch has CRLF endings. I manually changed the patch's line endings under the versions/libpq folder and applied the same git apply command as vcpkg attempts:

C:/vcpkg/downloads/tools/git-2.26.2-1-windows/mingw32/bin/git.exe -c core.longpaths=true -c core.autocrlf=false --work-tree=. --git-dir=.git apply "C:/vcpkg/buildtrees/versioning/versions/libpq/04f39fd5eb9744d0d5a649098fe92281028d30f1/patches/windows/win_bison_flex.patch" --ignore-whitespace --whitespace=nowarn --verbose

and it was successfully applied.

So is there some way to preserve the line endings when the patches are replicated to the versioning directory?

[meastp]

@ras0219-msft This solved neither my build issue with libpq (when I tried to remove autocrlf false) nor the first part of abi_info. Could you please point to where this hash is computed? I'd like to have debug output for a single package to try to pinpoint the difference...

Should I open a new issue?

[PhoebeHui]

@meastp, I reopen this issue, so you don't need to open a new issue.

[meastp]

@JackBoosY Not really - see this comment for what's missing: #16615 (comment)

[PhoebeHui]

The new triplet option VCPKG_DISABLE_COMPILER_TRACKING would be used to disable compiler tracking, and it should be included in next release.

Related microsoft/vcpkg-tool@5e55749

[meastp]

@PhoebeHui Isn't that for the compiler? Does it turn the entire abi_tracking off? What I need is the tools version to either be ignored or (ideally) forced to always downloaded through vcpkg - I still want the compiler version to be tracked.

[Be-ing]

I am also stumbling over this issue. vcpkg is using the version of cmake from the outside environment in the ABI info rather than the cmake version downloaded to vcpkg/downloads/tools.

[Be-ing]

I found the problem. vcpkg searches for tools from the system before downloading them. If the system tool version >= the minimum required version, vcpkg does not use the cmake version it downloads:
https://github.com/microsoft/vcpkg-tool/blob/cb1e40d6fe1a97ec4e29b79c3a0b4bcb02acc532/src/vcpkg/tools.cpp#L240

There is the VCPKG_FORCE_SYSTEM_BINARIES environment variable, but that does the opposite of what I want. I want vcpkg to only use the fetched version of cmake.

[Be-ing]

Wait... debug output does show that vcpkg uses the version of cmake it downloads?? Then why is that not used in the ABI info?

-- Installing port from location: /Users/be/sw/mixxx/vcpkg/overlay/osx/qt5-base
[DEBUG] Found path: /usr/local/bin/ninja
[DEBUG] popen(/Users/be/sw/mixxx/vcpkg/downloads/tools/ninja-1.10.2-osx/ninja --version 2>&1)
[DEBUG] cmd_execute_and_stream_data() returned 0 after    38586 us
[DEBUG] Found path: /usr/bin/git
[DEBUG] popen(/usr/bin/git --version 2>&1)
[DEBUG] cmd_execute_and_stream_data() returned 0 after    21592 us
[DEBUG] popen(/Users/be/sw/mixxx/vcpkg/downloads/tools/cmake-3.21.1-osx/cmake-3.21.1-macos-universal/CMake.app/Contents/bin/cmake -DALL_FEATURES= -DCURRENT_PORT_DIR=/Users/be/sw/mixxx/vcpkg/overlay/osx/qt5-base -D_HOST_TRIPLET=x64-osx -DFEATURES=core -DPORT=qt5-base -DVCPKG_USE_HEAD_VERSION=0 -D_VCPKG_DOWNLOAD_TOOL=BUILT_IN -D_VCPKG_EDITABLE=0 -D_VCPKG_NO_DOWNLOADS=0 -DCMD=BUILD -DDOWNLOADS=/Users/be/sw/mixxx/vcpkg/downloads -DTARGET_TRIPLET=x64-osx -DTARGET_TRIPLET_FILE=/Users/be/sw/mixxx/vcpkg/overlay/triplets/x64-osx.cmake -DVCPKG_BASE_VERSION=2021-08-03 -DVCPKG_CONCURRENCY=5 -DVCPKG_PLATFORM_TOOLSET=external -DGIT=/usr/bin/git -DVCPKG_ROOT_DIR=/Users/be/sw/mixxx/vcpkg -DPACKAGES_DIR=/Users/be/sw/mixxx/vcpkg/packages -DBUILDTREES_DIR=/Users/be/sw/mixxx/vcpkg/buildtrees -D_VCPKG_INSTALLED_DIR=/Users/be/sw/mixxx/build/vcpkg_installed -DDOWNLOADS=/Users/be/sw/mixxx/vcpkg/downloads -DVCPKG_MANIFEST_INSTALL=OFF -P /Users/be/sw/mixxx/vcpkg/scripts/ports.cmake 2>&1

[Be-ing]

Oh, nevermind, I got myself into a weird state by deleting vcpkg/downloads... 🙃 Not sure why I did that. vcpkg does use the version of cmake it downloads if I don't interfere with it 😆

brew install cmake # installs CMake 3.21.3 currently
rm -rf vcpkg/vcpkg vcpkg/download build
# vcpkg bootstrap downloads CMake 3.21.1, matching the CI server
cmake -D VCPKG_INSTALL_OPTIONS="--debug" -S . -B build

Now it works.

[JackBoosY]

@BillyONeal Was this issue fixed in the latest master?

[BillyONeal]

@BillyONeal Was this issue fixed in the latest master?

I don't believe anything has changed here. I know microsoft/vcpkg-tool#181 touches a similar area.

[Be-ing]

@JoergAtGithub seems to be dealing with the same or similar issue on Windows with vcpkg on GitHub Actions using CMake 3.21.3 but locally vcpkg uses the copy of CMake 3.21.1 that it downloads.

[Be-ing]

Because the CMake version is included in the cache key, I think vcpkg should unconditionally use the version of CMake it downloads and error if this is not found. I do not think it should fall back to the system CMake.

[BillyONeal]

Because the CMake version is included in the cache key, I think vcpkg should unconditionally use the version of CMake it downloads and error if this is not found. I do not think it should fall back to the system CMake.

I still believe that always using a downloaded copy is incorrect; large initial downloads like that to use anything have been a consistent end user complaint which is why we try to use the system copy if it is new enough.

However, we should be using the same version number we use from the system in the cache key, not whatever version we were looking for. If we aren't doing that it is indeed a bug.

[Be-ing]

large initial downloads like that to use anything have been a consistent end user complaint which is why we try to use the system copy if it is new enough

I understand why users would be annoyed with that... OTOH, for my uses the tools vcpkg downloads have a trivial size compared to the source code of the packages and the build artifacts. By contrast the cost of cache misses is waiting over 4 hours for a local build (for context, here is the vcpkg.json file).

we should be using the same version number we use from the system in the cache key

But this is the cause of cache misses. So what should users do to make use of a binary cache from a server for local development, manually install the exact same CMake version that the server uses and repeat that whenever the server gets updated? This is quite cumbersome and I don't want to ask contributors to go through that trouble.

I am unclear why the CMake version is included in the cache key in the first place. Do different versions of CMake really produce ABI incompatible libraries?

The binary caching feature of vcpkg is currently quite frustrating to use. First a user has to find out that a hash of the compiler executable is included in the cache key and find the documentation to disable that. Now there is this issue with mismatched CMake versions. It's a lot of trouble, and a lot time, to figure out how to get vcpkg's binary caching to work consistently. I understand vcpkg aims for reproducible builds as much as it can, but I am in doubt this should come at the cost of usability.

[Be-ing]

I still believe that always using a downloaded copy is incorrect; large initial downloads like that to use anything have been a consistent end user complaint which is why we try to use the system copy if it is new enough.

In general, this is good. Checking for a minimum required version of a tool and using the system's copy of the tool is appropriate in most cases. But it is not appropriate when the version of the tool is included in the cache key. IMO either vcpkg should unconditionally download its own copy of CMake or the CMake version should not be used in the cache key.

[Be-ing]

I am willing to fix this myself, but I would need to know how it should be fixed. Either the CMake version could be removed from the cache key or vcpkg could always download & use its own copy of CMake with a specific version. If you are certain that the default behavior should remain as it is, this fix could be optional, but I think it should be the default.

[meastp]

@Be-ing Your proposed fix (either og them) would save us a lot of unnecessary hassle and build time. Thank you for this :)

[Be-ing]

I made a PR for vcpkg-tool to require an exact version match for CMake so vcpkg will always download its own copy of CMake if the system version is not an exact match. That was surprisingly easy to implement. The vcpkg-tool C++ code was easy to understand (apart from the wide usage of "maybe"). :)

[Be-ing]

An alternative solution could be treating the CMake version in the binary cache key as only the minimum required version but not putting the actual version of CMake used in the cache key. That would allow using newer system versions of CMake without needing to download an exact version of CMake. That feels hacky and fragile though; allowing a range of versions to be compatible with the ABI info seems to kinda work against the purpose of tracking the ABI info. If it is permissible to not have an exact match of the CMake version to have a cache hit, then I think it would make more sense to not track the CMake version in the ABI info.

[Be-ing]

Apparently GitHub Actions updated its version of PowerShell from version 7.1.4 to 7.1.5 and the PowerShell version is included in the ABI info so that triggered a full rebuild of Windows libraries. So I made vcpkg require an exact version of PowerShell as well as CMake in microsoft/vcpkg-tool#223

[Be-ing]

This is finally fixed with microsoft/vcpkg-tool#234 and #21471. You need to use the new --x-abi-tools-use-exact-versions command line option in addition to disabling compiler tracking. So at the top of your CMakeLists.txt, before project():

set(VCPKG_FEATURE_FLAGS "-compilertracking,manifests,registries,versions")
set(VCPKG_INSTALL_OPTIONS "--x-abi-tools-use-exact-versions")

[meastp]

@Be-ing Great, thanks! I didn't find documentation for the two features. Do I need to use the two in combination, or can I keep compiler tracking (I'm assuming this means the exact compiler version/update is ignored?) while using consistent versions of tools?

[ras0219-msft]

Yes, they should function independently so you can use one or the other (or both or neither). For changing compiler tracking, I recommend using the documented triplet variable VCPKG_DISABLE_COMPILER_TRACKING 1 instead of the experimental-by-design feature flags approach.

[Be-ing]

I was thinking these options should be combined into one, so I'm curious what kind of setup you have where including the compiler in the abi_info is desired but you need vcpkg to use exact tool versions?

[meastp]

Thanks,both :)

@Be-ing Well, the primary motivation for the tools feature is to be able to share binary caching between the Azure build agent and local developer machines. I still would like to trigger a rebuild when the compiler revision is changed. Since the build agent is the one that gets new compiler versions first (it's a Microsoft-Hosted Agent, so it is usually updated switfly), rebuilding do not affect us too much, and the binarycache is ready when devs upgrade their local compiler version.

How do you trigger a rebuild of the packages in case of a compiler bug? Perhaps VCPKG_DISABLE_COMPILER_TRACKING instead should be a version number that can be increased from 1 in case of this happening? Or have you solved this some other way?

[ras0219-msft]

We do hash the triplet file itself, so you can manually force a rebuild by changing a comment in the file. However, I generally would recommend against VCPKG_DISABLE_COMPILER_TRACKING if possible.