Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow shallow git registries #37

Merged
merged 1 commit into from
Mar 26, 2021

Conversation

dg0yt
Copy link
Contributor

@dg0yt dg0yt commented Mar 24, 2021

git fetch normally refuses to deal with shallow repositories.
For vcpkg, this makes it impossible to use shallow clones of remote
registries such as VCPKG_INSTALLATION_ROOT on Azure Pipelines.
An extra parameter makes git fetch accept shallow repositories.

'git fetch' normally refuses to deal with shallow repositories.
For vcpkg, this makes it impossible to use shallow clones of remote
registries such as 'VCPKG_INSTALLATION_ROOT' on Azure Pipelines.
An extra parameter makes 'git fetch' accept shallow repositories.
@dg0yt dg0yt force-pushed the shallow-registries branch from 51ec8dc to 58a0678 Compare March 24, 2021 07:53
git_cmd_builder(*this, dot_git_dir, work_tree).string_arg("fetch").string_arg("--").string_arg(repo);
System::Command fetch_git_ref = git_cmd_builder(*this, dot_git_dir, work_tree)
.string_arg("fetch")
.string_arg("--update-shallow")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.string_arg("--update-shallow")
.string_arg("--unshallow")

Since git repos reference tree objects in the history, we need the full history available.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point is: For a shallow clone such as VCPK_INSTALLATION_ROOT in Azure Pipelines, one would need to unshallow the clone at VCPKG_INSTALLATION_ROOT. You cannot apply --unshallow when fetching from the shallow clone.

Since git repos reference tree objects in the history, we need the full history available.

I'm not a git tree object expert. Anyway, do you really always need the full history, or just enough depth to resolve the references for the desired ports and versions?

@dg0yt
Copy link
Contributor Author

dg0yt commented Mar 24, 2021

FWIW, the proposed patch works for me with VCPK_INSTALLATION_ROOT in Azure Pipelines and three selected packages ("vcpkg-cmake", "vcpkg-cmake-config", "libiconv"). I understand that this will break when I build a port that demands an older version which is not in the shallow history, and I did not test what kind of error reporting I can expect in this case.
The behaviour before this patch is git's refusal to fetch, and vcpkg hiding the git fetch error message.

A full successful build of #35 + #37 is at
https://dev.azure.com/dg0yt/superbuild/_build/results?buildId=170

A failed attempt with VCPKG_INSTALLATION_ROOT is
https://dev.azure.com/dg0yt/superbuild/_build/results?buildId=160:

2021-03-22T21:03:32.6785770Z + /Users/runner/work/1/ports/vcpkg --debug search dpkg
2021-03-22T21:03:32.6787590Z ===
2021-03-22T21:03:32.7400660Z [DEBUG] Feature flag 'binarycaching' unset
2021-03-22T21:03:32.7470880Z [DEBUG] Feature flag 'manifests' unset
2021-03-22T21:03:32.7573670Z [DEBUG] Feature flag 'compilertracking' unset
2021-03-22T21:03:32.7598220Z [DEBUG] Feature flag 'registries' = on
2021-03-22T21:03:32.7610180Z [DEBUG] Feature flag 'versions' unset
2021-03-22T21:03:32.7611940Z [DEBUG] BuiltinRegistry initialized with: ""
2021-03-22T21:03:32.7613580Z [DEBUG] Using vcpkg-root: /Users/runner/work/1/ports
2021-03-22T21:03:32.7615260Z [DEBUG] Using installed-root: /Users/runner/work/1/ports/installed
2021-03-22T21:03:32.7616470Z [DEBUG] BuiltinRegistry initialized with: ""
2021-03-22T21:03:32.7618080Z [DEBUG] Using buildtrees-root: /Users/runner/work/1/ports/buildtrees
2021-03-22T21:03:32.7619800Z [DEBUG] Using downloads-root: /Users/runner/work/1/ports/$(Build.BuildDirectory)/downloads
2021-03-22T21:03:32.7621570Z [DEBUG] Using packages-root: /Users/runner/work/1/ports/packages
2021-03-22T21:03:32.7623210Z [DEBUG] Using scripts-root: /Users/runner/work/1/ports/scripts
2021-03-22T21:03:32.7624830Z [DEBUG] Using ports-root: /Users/runner/work/1/ports/ports
2021-03-22T21:03:32.7626460Z [DEBUG] Using versions-root: /Users/runner/work/1/ports/versions
2021-03-22T21:03:32.7627310Z [DEBUG] Found path: /usr/local/bin/git
2021-03-22T21:03:32.7628520Z [DEBUG] Found path: /usr/bin/git
2021-03-22T21:03:32.7630390Z [DEBUG] popen(/usr/local/bin/git --version 2>&1)
2021-03-22T21:03:32.7631650Z [DEBUG] cmd_execute_and_stream_data() returned 0 after    17216 us
2021-03-22T21:03:32.7633150Z [DEBUG] popen(/usr/local/bin/git --git-dir=/Users/runner/.cache/vcpkg/registries/git/.git --work-tree=/Users/runner/.cache/vcpkg/registries/git init 2>&1)
2021-03-22T21:03:32.7937240Z [DEBUG] cmd_execute_and_stream_data() returned 0 after    36189 us
2021-03-22T21:03:32.7940860Z [DEBUG] popen(/usr/local/bin/git --git-dir=/Users/runner/.cache/vcpkg/registries/git/.git --work-tree=/Users/runner/.cache/vcpkg/registries/git fetch -- file:///usr/local/share/vcpkg 2>&1)
2021-03-22T21:03:33.0422300Z [DEBUG] cmd_execute_and_stream_data() returned 0 after   248667 us
2021-03-22T21:03:33.0425090Z [DEBUG] popen(/usr/local/bin/git --git-dir=/Users/runner/.cache/vcpkg/registries/git/.git --work-tree=/Users/runner/.cache/vcpkg/registries/git rev-parse FETCH_HEAD 2>&1)
2021-03-22T21:03:33.0526490Z [DEBUG] cmd_execute_and_stream_data() returned 32768 after     9968 us
2021-03-22T21:03:33.0528850Z Error: Failed to fetch from remote registry `file:///usr/local/share/vcpkg`: Error: Failed to rev-parse FETCH_HEAD.
2021-03-22T21:03:33.0530760Z fatal: ambiguous argument 'FETCH_HEAD': unknown revision or path not in the working tree.
2021-03-22T21:03:33.0532480Z Use '--' to separate paths from revisions, like this:
2021-03-22T21:03:33.0534070Z 'git <command> [<revision>...] -- [<file>...]'
2021-03-22T21:03:33.0539590Z FETCH_HEAD
2021-03-22T21:03:33.0566910Z 
2021-03-22T21:03:33.0584200Z 
2021-03-22T21:03:33.0595610Z [DEBUG] /Users/runner/work/1/vcpkg-tool/src/vcpkg/registries.cpp(60)
2021-03-22T21:03:33.0627170Z [DEBUG] Exiting after 372923 us (366972 us)

@dg0yt
Copy link
Contributor Author

dg0yt commented Mar 26, 2021

I don't have a strong opinion about whether shallow repository support is really needed. But at least the error reporting could be improved.

For Azure Pipelines, the repository at VCPKG_INSTALLATION_ROOT is provided with depth 1. I suppose this won't be enough most of the time.
https://github.com/actions/virtual-environments/blob/6168b56552e5c4f6e4d884865e5f2454a8a4f94b/images/macos/provision/core/vcpkg.sh#L9

However, AFAIU, it cannot be used as a filesystem registry, because it provides "git-tree" fields, no "path" fields.

@ras0219-msft
Copy link
Contributor

ras0219-msft commented Mar 26, 2021

I see, sorry, I misunderstood the situation!

To restate for future readers: you're using a pre-cloned, shallow repo as a git registry. Therefore, when we attempt to fetch the registry into the local registry git, fetch fails because it doesn't want to fetch from a shallow registry (the current registry is importantly not shallow). Therefore, --unshallow is meaningless here because the local registry is not shallow.

This solution LGTM given the problem! Thanks!

@ras0219-msft ras0219-msft merged commit 1b10bed into microsoft:main Mar 26, 2021
strega-nil added a commit that referenced this pull request Apr 15, 2021
The following PRs are included:

* hopefully fix crash in constraints (#60)
* [vcpkg] allow --version to check the version (#50)
* Remove baseline warning (#27)
* [git] always pass autocrlf=false (#58)
* ignore QtCreator CMake project files (#54)
* ignore .DS_store files (#53)
* [vcpkg] x-add-version now also checks if the manifest file is properly formatted (#43)
* hopefully fix ci issue #16773 (#34)
* Add docs to set VCPKG_ROOT to run tests (#45)
* [vcpkg] x-add-version improve speed by calling get_builtin_baseline only once (#44)
* add clang-format version to format-cxxcode (#41)
* [vcpkg] Introduce experimental workaround X_VCPKG_NUGET_ID_PREFIX (#40)
* [supports] Add `native` identifier expression and x-check-support command (#29)
* [metrics] Split reporting of installs into name:triplet (#39)
* [vcpkg] Improve error when accessing missing feature (#38)
* [vcpkg] Allow shallow git registries (#37)
* Disable git autocrlf when archiving tree (#36)
* Use only named packages from extra registries (#35)
* [registries] add metrics (#30)
* Add vcpkg policy cmake helper port support (#17)
* [osx] add support for rosetta (#23)
* don't build tls12-download unless it's needed (#33)
* Add new telemetry points for versioning (#21)
* add cmake_minimum_required to vcpkg_tags (#25)
* [x-add-versions] Perform atomic replacement of versioning files (#28)
* [tools] support gsutil (#19)
* add CUDA 11.1 and 11.2 to KEEP_ENV_VARS defaults (#26)
* Add finite timeout on CURL metrics endpoint. (#22)
* fix UB in make_error_code(utf8_errc) (#18)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants