[vcpkg] Add feature option 'withMirror' ⚡ #12532

JackBoosY · 2020-07-22T07:38:17Z

This PR will add the use of mirror download address function, allow users to set the mirror address of ports and tools through parameters when building vcpkg to reduce download time.

usage:
To use vcpkg mirror:
set environment VCPKG_EXPERIMENTAL_MIRROR_URL to mirror address, example: 139.196.94.253.

To download all ports to mirror machine:

set VCPKG_EXPERIMENTAL_MIRROR_URL PATH_TO_DOWNLOADS
./vcpkg x-mirror

Thanks @ras0219-msft for the advice.

Closes #11040

BillyONeal · 2020-07-23T21:39:54Z

Thanks, this is great 👍

I would like to see the control being done with an environment variable and/or command line option rather than compiling it into the tool with bootstrap, because once we start doing binary releases that won't work anymore.

…e rename to full hash value

JackBoosY · 2020-07-24T07:44:03Z

toolsrc/src/vcpkg/base/downloads.cpp

@@ -12,134 +12,6 @@

 namespace vcpkg::Downloads
 {
-#if defined(_WIN32)
-    static void winhttp_download_file(Files::Filesystem& fs,


Since winhttp.h has conflict with wininet.h, I split this function into two files.

JackBoosY · 2020-08-25T08:36:23Z

qt5-tools:

Qt5Guid.lib(qrhid3d11.obj) : error LNK2001: unresolved external symbol IID_IDXGIFactory2
Qt5Guid.lib(qrhid3d11.obj) : error LNK2001: unresolved external symbol IID_ID3D11DeviceContext1
Qt5Guid.lib(qrhid3d11.obj) : error LNK2001: unresolved external symbol IID_ID3DUserDefinedAnnotation

JackBoosY · 2020-08-25T09:04:39Z

clapack:

Building package clapack[core]:x64-osx... done
Installing package clapack[core]:x64-osx...
The following files are already installed in /Users/vagrant/Data/installed/x64-osx and are in conflict with clapack:x64-osx

Installed by lapack-reference:x64-osx
    lib/liblapack.a

JackBoosY · 2020-08-26T03:09:51Z

qt5-tools:

Qt5Guid.lib(qrhid3d11.obj) : error LNK2001: unresolved external symbol IID_IDXGIFactory2
Qt5Guid.lib(qrhid3d11.obj) : error LNK2001: unresolved external symbol IID_ID3D11DeviceContext1
Qt5Guid.lib(qrhid3d11.obj) : error LNK2001: unresolved external symbol IID_ID3DUserDefinedAnnotation

@Neumann-A It seems that qt5-tools forgot to link to dxguid.lib?

ras0219 · 2020-09-01T04:42:46Z

scripts/cmake/vcpkg_download_distfile.cmake

+
+    if (ENV{VCPKG_EXPERIMENTAL_MIRROR_URL})
+        message(STATUS "Use vcpkg mirror $ENV{VCPKG_EXPERIMENTAL_MIRROR_URL}")
+        set(vcpkg_download_distfile_URLS "ftp://$ENV{VCPKG_EXPERIMENTAL_MIRROR_URL}/${vcpkg_download_distfile_SHA512}")


This should do something reasonable in the vcpkg_download_distfile_SKIP_SHA512 case

Without SHA512, vcpkg_download_distfile cannot splice the download URL.

BillyONeal

I will merge this for you after #13038 lands; they both touch a lot of the same stuff but this one is a bit smaller so merging will be less painful.

Do we have a spec for this feature? As I originally understood it this was the ability to redirect our download attempts somewhere else; this "crawl" attempt thing is something very different. I don't know if that changed from when I looked last or if I was just blind due to being delirious trying to ship jthread in STL land.

BillyONeal · 2020-09-01T04:41:24Z

toolsrc/include/vcpkg/vcpkgcmdarguments.h

@@ -154,6 +154,10 @@ namespace vcpkg
        constexpr static StringLiteral WAIT_FOR_LOCK_SWITCH = "x-wait-for-lock";
        Optional<bool> wait_for_lock = nullopt;

+        constexpr static StringLiteral VCPKG_EXPERIMENTAL_MIRROR_URL = "VCPKG_EXPERIMENTAL_MIRROR_URL";


I don't think we should name the variable EXPERIMENTAL because someday it won't be and there's no reason to force everyone to edit their scripts then?

OTOH we can continue respecting the experimental form for backwards compatibility and this makes it clear for users that they are using an unsupported feature. Alternatively, the tool could emit a big yelllow warning at the top every time it's run with this env var set.

toolsrc/src/vcpkg/vcpkgpaths.cpp

toolsrc/src/vcpkg/commands.mirror.cpp

toolsrc/src/vcpkg/commands.cpp

ras0219 · 2020-09-01T04:46:55Z

toolsrc/include/vcpkg/vcpkgcmdarguments.h

@@ -154,6 +154,10 @@ namespace vcpkg
        constexpr static StringLiteral WAIT_FOR_LOCK_SWITCH = "x-wait-for-lock";
        Optional<bool> wait_for_lock = nullopt;

+        constexpr static StringLiteral VCPKG_EXPERIMENTAL_MIRROR_URL = "VCPKG_EXPERIMENTAL_MIRROR_URL";


We conventionally use lowercase here

Suggested change

constexpr static StringLiteral VCPKG_EXPERIMENTAL_MIRROR_URL = "VCPKG_EXPERIMENTAL_MIRROR_URL";

constexpr static StringLiteral VCPKG_EXPERIMENTAL_MIRROR_URL = "x-mirror-url";

The environment variable name is set here. Is this correct?

toolsrc/src/vcpkg/commands.mirror.cpp

toolsrc/src/vcpkg/base/downloads.cpp

ras0219 · 2020-09-01T04:52:47Z

toolsrc/src/vcpkg/tools.cpp

+        if (paths.vcpkg_use_mirror)
+        {
+            url = "ftp://";
+            url += paths.vcpkg_mirror_url;


If vcpkg_mirror_url is changed to include the protocol, this would enable https mirrors as well and generally be more robust

Additionally, this should arguably be moved into download_file() so that every case across vcpkg uses it.

I am currently unable to change it to https, can we make these changes later?

toolsrc/include/vcpkg/commands.mirror.h

toolsrc/include/vcpkg/base/downloads.h

JackBoosY · 2020-09-01T08:21:36Z

Since the source package downloaded using x-mirror will be automatically renamed to sha512 value, do we need to provide the list of downloaded source packages to the mirror maintainer?

…jack/vcpkg_mirror

JackBoosY · 2020-09-11T02:12:39Z

@ras0219 @ras0219-msft Please review again.

…jack/vcpkg_mirror

soroshsabz · 2021-05-07T16:50:40Z

@JackBoosY why this PR closed? Where this functionality exist?

Thanks

BillyONeal · 2021-05-07T17:06:14Z

@JackBoosY why this PR closed? Where this functionality exist?

Thanks

It needs to be replayed over in vcpkg-tool instead at a minimum

JackBoosY · 2021-05-08T02:52:41Z

This PR was replaced by #13639.

[VCPKG] Add feature mirror

a4cc356

JackBoosY added category:vcpkg-feature The issue is a new capability of the tool that doesn’t already exist and we haven’t committed info:internal This PR or Issue was filed by the vcpkg team. labels Jul 22, 2020

JackBoosY changed the title ~~[VCPKG] Add feature mirror~~ [VCPKG] Add feature option 'useMirror' Jul 22, 2020

JackBoosY added 6 commits July 22, 2020 01:50

Correct build script, add mirror addr to cmake args

2b3e855

clang-format

d7cd136

suitable mirror in cmake file, add code in vcpkg internal, fix vcxproj

881fbe0

Correct shell

d17ae85

Correct shell

874a6bd

Add macro USE_MIRROR and VCPKG_MIRROR to CMakeLists.txt

6b92077

JackBoosY changed the title ~~[VCPKG] Add feature option 'useMirror'~~ [VCPKG] Add feature option 'withMirror' Jul 23, 2020

JackBoosY added 2 commits July 22, 2020 23:19

Change the tools download address to the mirror address

d108534

Add mirror command

849377c

JackBoosY requested review from strega-nil and ras0219-msft and removed request for strega-nil July 23, 2020 09:17

JackBoosY assigned PhoebeHui Jul 23, 2020

JackBoosY changed the title ~~[VCPKG] Add feature option 'withMirror'~~ [vcpkg] Add feature option 'withMirror':zap: Jul 23, 2020

JackBoosY changed the title ~~[vcpkg] Add feature option 'withMirror':zap:~~ [vcpkg] Add feature option 'withMirror' ⚡ Jul 23, 2020

JackBoosY changed the title ~~[vcpkg] Add feature option 'withMirror' ⚡~~ [vcpkg] Add feature option 'withMirror' ⚡ Jul 23, 2020

PhoebeHui mentioned this pull request Jul 24, 2020

[Request] Local repository #5119

Closed

JackBoosY added 2 commits July 23, 2020 21:02

Switch build option to environment VCPKG_EXPERIMENTAL_MIRROR_URL, fil…

f04daea

…e rename to full hash value

remove useless macro

83fe7f9

JackBoosY marked this pull request as ready for review July 24, 2020 05:38

JackBoosY marked this pull request as draft July 24, 2020 05:38

improve environment code

f4e3e42

JackBoosY commented Jul 24, 2020

View reviewed changes

clang-format

e34006a

JackBoosY mentioned this pull request Aug 25, 2020

[bigint/zxing-cpp] Adapter xcode 5.1.1 and set it as a dependency of zxing-cpp #13129

Merged

JackBoosY mentioned this pull request Aug 26, 2020

[baseline] Skip check of clapack:x64-osx in the baseline #13148

Merged

ras0219 reviewed Sep 1, 2020

View reviewed changes

BillyONeal requested changes Sep 1, 2020

View reviewed changes

ras0219 suggested changes Sep 1, 2020

View reviewed changes

JackBoosY added 4 commits September 1, 2020 00:14

Rework commands.mirror

5c23abb

Reduce include files

d26bcde

Reduce include files

f5843c4

Remove vcpkg_use_mirror, fix VCPKG_EXPERIMENTAL_MIRROR_URL judgment

942b8dd

JackBoosY added 5 commits September 1, 2020 01:29

Re-fix VCPKG_EXPERIMENTAL_MIRROR_URL judgment

1ca45af

clang-format

9edf845

clang-format

d9313de

Merge branch 'master' of https://github.com/microsoft/vcpkg into dev/…

940f46c

…jack/vcpkg_mirror

Merge master branch

e190ce6

JackBoosY requested a review from BillyONeal September 9, 2020 05:33

JackBoosY added 2 commits September 17, 2020 19:57

Merge branch 'master' of https://github.com/microsoft/vcpkg into dev/…

cf59c08

…jack/vcpkg_mirror

Restore vcpkg.vcxporj

a306414

BillyONeal mentioned this pull request Sep 21, 2020

[vcpkg] Add experimental $X_VCPKG_ASSET_SOURCES for source caching #13639

Merged

2 tasks

JackBoosY mentioned this pull request Sep 24, 2020

I want to specify a downloader for VCPKG. What should I do #13690

Closed

JackBoosY closed this Oct 9, 2020

JackBoosY deleted the dev/jack/vcpkg_mirror branch May 26, 2022 06:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vcpkg] Add feature option 'withMirror' ⚡ #12532

[vcpkg] Add feature option 'withMirror' ⚡ #12532

JackBoosY commented Jul 22, 2020 •

edited

Loading

BillyONeal commented Jul 23, 2020

JackBoosY Jul 24, 2020

JackBoosY commented Aug 25, 2020

JackBoosY commented Aug 25, 2020 •

edited

Loading

JackBoosY commented Aug 26, 2020

ras0219 Sep 1, 2020

JackBoosY Sep 1, 2020

BillyONeal left a comment

BillyONeal Sep 1, 2020

ras0219 Sep 1, 2020

ras0219 Sep 1, 2020

JackBoosY Sep 1, 2020

ras0219 Sep 1, 2020

ras0219 Sep 1, 2020

JackBoosY Sep 1, 2020

JackBoosY commented Sep 1, 2020

JackBoosY commented Sep 11, 2020

soroshsabz commented May 7, 2021

BillyONeal commented May 7, 2021

JackBoosY commented May 8, 2021

	constexpr static StringLiteral VCPKG_EXPERIMENTAL_MIRROR_URL = "VCPKG_EXPERIMENTAL_MIRROR_URL";
	constexpr static StringLiteral VCPKG_EXPERIMENTAL_MIRROR_URL = "x-mirror-url";

[vcpkg] Add feature option 'withMirror' ⚡ #12532

[vcpkg] Add feature option 'withMirror' ⚡ #12532

Conversation

JackBoosY commented Jul 22, 2020 • edited Loading

BillyONeal commented Jul 23, 2020

Choose a reason for hiding this comment

JackBoosY commented Aug 25, 2020

JackBoosY commented Aug 25, 2020 • edited Loading

JackBoosY commented Aug 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BillyONeal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JackBoosY commented Sep 1, 2020

JackBoosY commented Sep 11, 2020

soroshsabz commented May 7, 2021

BillyONeal commented May 7, 2021

JackBoosY commented May 8, 2021

JackBoosY commented Jul 22, 2020 •

edited

Loading

JackBoosY commented Aug 25, 2020 •

edited

Loading