Skip to content

Commit

Permalink
[vcpkg] Implement --editable (#12200)
Browse files Browse the repository at this point in the history
--editable suppresses binary caching and source re-extraction for packages listed on the command line (similar to --head).

This fundamentally changes the port maintenance loop, so several example documents were updated. To avoid users having substantial changes suddenly destroyed by forgetting to pass --editable, "clean" sources have different extract locations.

The undocumented command `build` implies `--editable`.

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
ras0219 and ras0219-msft authored Jul 7, 2020

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
1 parent 12333a6 commit f9bdf13
Showing 8 changed files with 163 additions and 130 deletions.
58 changes: 29 additions & 29 deletions docs/examples/installing-and-using-packages.md
Original file line number Diff line number Diff line change
@@ -14,7 +14,7 @@ First, we need to know what name [SQLite](https://sqlite.org) goes by in the por
```no-highlight
PS D:\src\vcpkg> .\vcpkg search sqlite
libodb-sqlite 2.4.0 Sqlite support for the ODB ORM library
sqlite3 3.15.0 SQLite is a software library that implements a se...
sqlite3 3.32.1 SQLite is a software library that implements a se...
If your library is not listed, please open an issue at:
https://github.com/Microsoft/vcpkg/issues
@@ -24,38 +24,38 @@ Looking at the list, we can see that the port is named "sqlite3". You can also r
Installing is then as simple as using the `install` command.
```no-highlight
PS D:\src\vcpkg> .\vcpkg install sqlite3
-- CURRENT_INSTALLED_DIR=D:/src/vcpkg/installed/x86-windows
-- DOWNLOADS=D:/src/vcpkg/downloads
-- CURRENT_PACKAGES_DIR=D:/src/vcpkg/packages/sqlite3_x86-windows
-- CURRENT_BUILDTREES_DIR=D:/src/vcpkg/buildtrees/sqlite3
-- CURRENT_PORT_DIR=D:/src/vcpkg/ports/sqlite3/.
-- Downloading https://sqlite.org/2016/sqlite-amalgamation-3150000.zip...
-- Downloading https://sqlite.org/2016/sqlite-amalgamation-3150000.zip... OK
-- Testing integrity of downloaded file...
-- Testing integrity of downloaded file... OK
-- Extracting source D:/src/vcpkg/downloads/sqlite-amalgamation-3150000.zip
-- Extracting done
-- Configuring x86-windows-rel
-- Configuring x86-windows-rel done
-- Configuring x86-windows-dbg
-- Configuring x86-windows-dbg done
-- Build x86-windows-rel
-- Build x86-windows-rel done
-- Build x86-windows-dbg
-- Build x86-windows-dbg done
-- Package x86-windows-rel
-- Package x86-windows-rel done
-- Package x86-windows-dbg
-- Package x86-windows-dbg done
Computing installation plan...
The following packages will be built and installed:
sqlite3[core]:x86-windows
Starting package 1/1: sqlite3:x86-windows
Building package sqlite3[core]:x86-windows...
-- Downloading https://sqlite.org/2020/sqlite-amalgamation-3320100.zip...
-- Extracting source C:/src/vcpkg/downloads/sqlite-amalgamation-3320100.zip
-- Applying patch fix-arm-uwp.patch
-- Using source at C:/src/vcpkg/buildtrees/sqlite3/src/3320100-15aeda126a.clean
-- Configuring x86-windows
-- Building x86-windows-dbg
-- Building x86-windows-rel
-- Performing post-build validation
-- Performing post-build validation done
Package sqlite3:x86-windows is installed
Building package sqlite3[core]:x86-windows... done
Installing package sqlite3[core]:x86-windows...
Installing package sqlite3[core]:x86-windows... done
Elapsed time for package sqlite3:x86-windows: 12 s
Total elapsed time: 12.04 s
The package sqlite3:x86-windows provides CMake targets:
find_package(sqlite3 CONFIG REQUIRED)
target_link_libraries(main PRIVATE sqlite3)
```

We can check that sqlite3 was successfully installed for x86 windows desktop by running the `list` command.
We can check that sqlite3 was successfully installed for x86 Windows desktop by running the `list` command.
```no-highlight
PS D:\src\vcpkg> .\vcpkg list
sqlite3:x86-windows 3.15.0 SQLite is a software library that implements a se...
sqlite3:x86-windows 3.32.1 SQLite is a software library that implements a se...
```

To install for other architectures and platforms such as Universal Windows Platform or x64 Desktop, you can suffix the package name with `:<target>`.
@@ -82,7 +82,7 @@ Installing new libraries will make them instantly available.
```
*Note: You will need to restart Visual Studio or perform a Build to update intellisense with the changes.*

You can now simply use File -> New Project in Visual Studio 2015 or Visual Studio 2017 and the library will be automatically available. For SQLite, you can try out their [C/C++ sample](https://sqlite.org/quickstart.html).
You can now simply use File -> New Project in Visual Studio and the library will be automatically available. For SQLite, you can try out their [C/C++ sample](https://sqlite.org/quickstart.html).

To remove the integration for your user, you can use `.\vcpkg integrate remove`.

@@ -92,7 +92,7 @@ To remove the integration for your user, you can use `.\vcpkg integrate remove`.
The best way to use installed libraries with cmake is via the toolchain file `scripts\buildsystems\vcpkg.cmake`. To use this file, you simply need to add it onto your CMake command line as:
`-DCMAKE_TOOLCHAIN_FILE=D:\src\vcpkg\scripts\buildsystems\vcpkg.cmake`.

If you are using CMake through Open Folder with Visual Studio 2017 you can define `CMAKE_TOOLCHAIN_FILE` by adding a "variables" section to each of your `CMakeSettings.json` configurations:
If you are using CMake through Open Folder with Visual Studio you can define `CMAKE_TOOLCHAIN_FILE` by adding a "variables" section to each of your `CMakeSettings.json` configurations:

```json
{
38 changes: 17 additions & 21 deletions docs/examples/packaging-zipfiles.md
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@
### Bootstrap with `create`
First, locate a globally accessible archive of the library's sources. Zip, gzip, and bzip are all supported. Strongly prefer official sources or mirrors over unofficial mirrors.

*Looking at zlib's website, the URL http://zlib.net/zlib1211.zip looks appropriate.*
*Looking at zlib's website, the URL http://zlib.net/zlib-1.2.11.tar.gz looks appropriate.*

Second, determine a suitable package name. This should be ASCII, lowercase, and recognizable to someone who knows the library's "human name". If the library is already packaged in another package manager, prefer that name.

@@ -16,7 +16,7 @@ Finally, if the server's name for the archive is not very descriptive (such as d
All this information can then be passed into the `create` command, which will download the sources and bootstrap the packaging process inside `ports\<packagename>`.

```no-highlight
PS D:\src\vcpkg> .\vcpkg create zlib2 http://zlib.net/zlib-1.2.11.tar.gz zlib-1.2.11.zip
PS D:\src\vcpkg> .\vcpkg create zlib2 http://zlib.net/zlib-1.2.11.tar.gz zlib1211.tar.gz
-- Generated portfile: D:/src/vcpkg/ports/zlib2/portfile.cmake
```

@@ -34,25 +34,21 @@ Description: A Massively Spiffy Yet Delicately Unobtrusive Compression Library
The generated `portfile.cmake` will need some editing to correctly package most libraries in the wild, however we can start by trying out the build.

```no-highlight
PS D:\src\vcpkg> .\vcpkg build zlib2
-- CURRENT_INSTALLED_DIR=D:/src/vcpkg/installed/x86-windows
-- DOWNLOADS=D:/src/vcpkg/downloads
-- CURRENT_PACKAGES_DIR=D:/src/vcpkg/packages/zlib2_x86-windows
-- CURRENT_BUILDTREES_DIR=D:/src/vcpkg/buildtrees/zlib2
-- CURRENT_PORT_DIR=D:/src/vcpkg/ports/zlib2/.
-- Using cached D:/src/vcpkg/downloads/zlib-1.2.11.tar.gz
-- Testing integrity of cached file...
-- Testing integrity of cached file... OK
-- Extracting source D:/src/vcpkg/downloads/zlib-1.2.11.tar.gz
-- Extracting done
-- Configuring x86-windows-rel
-- Configuring x86-windows-rel done
-- Configuring x86-windows-dbg
-- Configuring x86-windows-dbg done
-- Build x86-windows-rel
-- Build x86-windows-rel done
-- Build x86-windows-dbg
-- Build x86-windows-dbg done
PS D:\src\vcpkg> .\vcpkg install zlib2
Computing installation plan...
The following packages will be built and installed:
zlib2[core]:x64-uwp
Starting package 1/1: zlib2:x64-uwp
Building package zlib2[core]:x64-uwp...
-- Using cached C:/src/vcpkg/downloads/zlib1211.tar.gz
-- Cleaning sources at C:/src/vcpkg/buildtrees/zlib2/src/1.2.11-deec42f53b.clean. Pass --editable to vcpkg to reuse sources.
-- Extracting source C:/src/vcpkg/downloads/zlib1211.tar.gz
-- Applying patch cmake_dont_build_more_than_needed.patch
-- Using source at C:/src/vcpkg/buildtrees/zlib2/src/1.2.11-deec42f53b.clean
-- Configuring x64-uwp
-- Building x64-uwp-dbg
-- Building x64-uwp-rel
-- Installing: C:/src/vcpkg/packages/zlib2_x64-uwp/share/zlib2/copyright
-- Performing post-build validation
Include files should not be duplicated into the /debug/include directory. If this cannot be disabled in the project cmake, use
file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/include)
126 changes: 65 additions & 61 deletions docs/examples/patching.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,29 @@
## Patching Example: Patching libpng to work for x86-uwp
## Patching Example: Patching libpng to work for x64-uwp

### Initial error logs
First, try building:

```no-highlight
PS D:\src\vcpkg> vcpkg install libpng:x86-uwp
-- CURRENT_INSTALLED_DIR=D:/src/vcpkg/installed/x86-uwp
-- DOWNLOADS=D:/src/vcpkg/downloads
-- CURRENT_PACKAGES_DIR=D:/src/vcpkg/packages/libpng_x86-uwp
-- CURRENT_BUILDTREES_DIR=D:/src/vcpkg/buildtrees/libpng
-- CURRENT_PORT_DIR=D:/src/vcpkg/ports/libpng/.
-- Using cached D:/src/vcpkg/downloads/libpng-1.6.24.tar.xz
-- Extracting done
-- Configuring x86-uwp-rel
-- Configuring x86-uwp-rel done
-- Configuring x86-uwp-dbg
-- Configuring x86-uwp-dbg done
-- Build x86-uwp-rel
PS D:\src\vcpkg> vcpkg install libpng:x64-uwp --editable
Computing installation plan...
The following packages will be built and installed:
libpng[core]:x64-uwp
Starting package 1/1: libpng:x64-uwp
Building package libpng[core]:x64-uwp...
-- Using cached D:/src/vcpkg/downloads/glennrp-libpng-v1.6.37.tar.gz
-- Extracting source D:/src/vcpkg/downloads/glennrp-libpng-v1.6.37.tar.gz
-- Using source at D:/src/vcpkg/buildtrees/libpng/src/v1.6.37-c993153cdf
-- Configuring x64-uwp
-- Building x64-uwp-rel
CMake Error at scripts/cmake/execute_required_process.cmake:14 (message):
Command failed: C:/Program
Files/CMake/bin/cmake.exe;--build;.;--config;Release
Command failed: C:/Program Files/CMake/bin/cmake.exe;--build;.;--config;Release
Working Directory: D:/src/vcpkg/buildtrees/libpng/x86-uwp-rel
Working Directory: D:/src/vcpkg/buildtrees/libpng/x64-uwp-rel
See logs for more information:
D:\src\vcpkg\buildtrees\libpng\build-x86-uwp-rel-out.log
D:\src\vcpkg\buildtrees\libpng\build-x86-uwp-rel-err.log
D:\src\vcpkg\buildtrees\libpng\build-x64-uwp-rel-out.log
D:\src\vcpkg\buildtrees\libpng\build-x64-uwp-rel-err.log
Call Stack (most recent call first):
scripts/cmake/vcpkg_build_cmake.cmake:3 (execute_required_process)
@@ -40,19 +37,19 @@ Error: build command failed
Next, looking at the above logs (build-...-out.log and build-...-err.log).

```no-highlight
// build-x86-uwp-rel-out.log
// build-x64-uwp-rel-out.log
...
"D:\src\vcpkg\buildtrees\libpng\x86-uwp-rel\ALL_BUILD.vcxproj" (default target) (1) ->
"D:\src\vcpkg\buildtrees\libpng\x86-uwp-rel\png.vcxproj" (default target) (3) ->
"D:\src\vcpkg\buildtrees\libpng\x64-uwp-rel\ALL_BUILD.vcxproj" (default target) (1) ->
"D:\src\vcpkg\buildtrees\libpng\x64-uwp-rel\png.vcxproj" (default target) (3) ->
(ClCompile target) ->
D:\src\vcpkg\buildtrees\libpng\src\libpng-1.6.24\pngerror.c(775): warning C4013: 'ExitProcess' undefined; assuming extern returning int [D:\src\vcpkg\buildtrees\libpng\x86-uwp-rel\png.vcxproj]
D:\src\vcpkg\buildtrees\libpng\src\v1.6.37-c993153cdf\pngerror.c(775): warning C4013: 'ExitProcess' undefined; assuming extern returning int [D:\src\vcpkg\buildtrees\libpng\x64-uwp-rel\png.vcxproj]
"D:\src\vcpkg\buildtrees\libpng\x86-uwp-rel\ALL_BUILD.vcxproj" (default target) (1) ->
"D:\src\vcpkg\buildtrees\libpng\x86-uwp-rel\png.vcxproj" (default target) (3) ->
"D:\src\vcpkg\buildtrees\libpng\x64-uwp-rel\ALL_BUILD.vcxproj" (default target) (1) ->
"D:\src\vcpkg\buildtrees\libpng\x64-uwp-rel\png.vcxproj" (default target) (3) ->
(Link target) ->
pngerror.obj : error LNK2019: unresolved external symbol _ExitProcess referenced in function _png_longjmp [D:\src\vcpkg\buildtrees\libpng\x86-uwp-rel\png.vcxproj]
D:\src\vcpkg\buildtrees\libpng\x86-uwp-rel\Release\libpng16.dll : fatal error LNK1120: 1 unresolved externals [D:\src\vcpkg\buildtrees\libpng\x86-uwp-rel\png.vcxproj]
pngerror.obj : error LNK2019: unresolved external symbol _ExitProcess referenced in function _png_longjmp [D:\src\vcpkg\buildtrees\libpng\x64-uwp-rel\png.vcxproj]
D:\src\vcpkg\buildtrees\libpng\x64-uwp-rel\Release\libpng16.dll : fatal error LNK1120: 1 unresolved externals [D:\src\vcpkg\buildtrees\libpng\x64-uwp-rel\png.vcxproj]
1 Warning(s)
2 Error(s)
@@ -65,7 +62,7 @@ Time Elapsed 00:00:04.19
Taking a look at [MSDN](https://msdn.microsoft.com/en-us/library/windows/desktop/ms682658(v=vs.85).aspx) shows that `ExitProcess` is only available for desktop apps. Additionally, it's useful to see the surrounding context:

```c
/* buildtrees\libpng\src\libpng-1.6.24\pngerror.c:769 */
/* buildtrees\libpng\src\v1.6.37-c993153cdf\pngerror.c:769 */
/* If control reaches this point, png_longjmp() must not return. The only
* choice is to terminate the whole process (or maybe the thread); to do
* this the ANSI-C abort() function is used unless a different method is
@@ -77,7 +74,7 @@ Taking a look at [MSDN](https://msdn.microsoft.com/en-us/library/windows/desktop

A recursive search for `PNG_ABORT` reveals the definition:
```no-highlight
PS D:\src\vcpkg\buildtrees\libpng\src\libpng-1.6.24> findstr /snipl "PNG_ABORT" *
PS D:\src\vcpkg\buildtrees\libpng\src\v1.6.37-c993153cdf> findstr /snipl "PNG_ABORT" *
CHANGES:701: Added PNG_SETJMP_SUPPORTED, PNG_SETJMP_NOT_SUPPORTED, and PNG_ABORT() macros
libpng-manual.txt:432:errors will result in a call to PNG_ABORT() which defaults to abort().
libpng-manual.txt:434:You can #define PNG_ABORT() to a function that does something
@@ -100,7 +97,7 @@ pngpriv.h:463:# define PNG_ABORT() abort()
This already gives us some great clues, but the full definition tells the complete story.

```c
/* buildtrees\libpng\src\libpng-1.6.24\pngpriv.h:459 */
/* buildtrees\libpng\src\v1.6.37-c993153cdf\pngpriv.h:459 */
#ifndef PNG_ABORT
# ifdef _WINDOWS_
# define PNG_ABORT() ExitProcess(0)
@@ -116,31 +113,31 @@ This already gives us some great clues, but the full definition tells the comple

We recommend using git to create the patch file, since you'll already have it installed.
```no-highlight
PS D:\src\vcpkg\buildtrees\libpng\src\libpng-1.6.24> git init .
Initialized empty Git repository in D:/src/vcpkg/buildtrees/libpng/src/libpng-1.6.24/.git/
PS D:\src\vcpkg\buildtrees\libpng\src\v1.6.37-c993153cdf> git init .
Initialized empty Git repository in D:/src/vcpkg/buildtrees/libpng/src/v1.6.37-c993153cdf/.git/
PS D:\src\vcpkg\buildtrees\libpng\src\libpng-1.6.24> git add .
PS D:\src\vcpkg\buildtrees\libpng\src\v1.6.37-c993153cdf> git add .
warning: LF will be replaced by CRLF in ANNOUNCE.
The file will have its original line endings in your working directory.
...
PS D:\src\vcpkg\buildtrees\libpng\src\libpng-1.6.24> git commit -m "temp"
PS D:\src\vcpkg\buildtrees\libpng\src\v1.6.37-c993153cdf> git commit -m "temp"
[master (root-commit) 68f253f] temp
422 files changed, 167717 insertions(+)
...
```

Now we can modify `pngpriv.h` to use `abort()` everywhere.
```c
/* buildtrees\libpng\src\libpng-1.6.24\pngpriv.h:459 */
/* buildtrees\libpng\src\v1.6.37-c993153cdf\pngpriv.h:459 */
#ifndef PNG_ABORT
# define PNG_ABORT() abort()
#endif
```

The output of `git diff` is already in patch format, so we just need to save the patch into the `ports/libpng` directory.
```no-highlight
PS buildtrees\libpng\src\libpng-1.6.24> git diff --ignore-space-at-eol | out-file -enc ascii ..\..\..\..\ports\libpng\use-abort-on-all-platforms.patch
PS buildtrees\libpng\src\v1.6.37-c993153cdf> git diff --ignore-space-at-eol | out-file -enc ascii ..\..\..\..\ports\libpng\use-abort-on-all-platforms.patch
```

Finally, we need to apply the patch after extracting the source.
@@ -163,42 +160,49 @@ vcpkg_configure_cmake(
To be completely sure this works from scratch, we need to remove the package and rebuild it:

```no-highlight
PS D:\src\vcpkg> vcpkg remove libpng:x86-uwp
Package libpng:x86-uwp was successfully removed
PS D:\src\vcpkg> vcpkg remove libpng:x64-uwp
Package libpng:x64-uwp was successfully removed
```
and complete delete the building directory: D:\src\vcpkg\buildtrees\libpng

Now we try a fresh, from scratch install.
```no-highlight
PS D:\src\vcpkg> vcpkg install libpng:x86-uwp
-- CURRENT_INSTALLED_DIR=D:/src/vcpkg/installed/x86-uwp
-- DOWNLOADS=D:/src/vcpkg/downloads
-- CURRENT_PACKAGES_DIR=D:/src/vcpkg/packages/libpng_x86-uwp
-- CURRENT_BUILDTREES_DIR=D:/src/vcpkg/buildtrees/libpng
-- CURRENT_PORT_DIR=D:/src/vcpkg/ports/libpng/.
-- Using cached D:/src/vcpkg/downloads/libpng-1.6.24.tar.xz
-- Extracting source D:/src/vcpkg/downloads/libpng-1.6.24.tar.xz
-- Extracting done
-- Configuring x86-uwp-rel
-- Configuring x86-uwp-rel done
-- Configuring x86-uwp-dbg
-- Configuring x86-uwp-dbg done
-- Build x86-uwp-rel
-- Build x86-uwp-rel done
-- Build x86-uwp-dbg
-- Build x86-uwp-dbg done
-- Package x86-uwp-rel
-- Package x86-uwp-rel done
-- Package x86-uwp-dbg
-- Package x86-uwp-dbg done
Package libpng:x86-uwp is installed
PS D:\src\vcpkg> vcpkg install libpng:x64-uwp
Computing installation plan...
The following packages will be built and installed:
libpng[core]:x64-uwp
Starting package 1/1: libpng:x64-uwp
Building package libpng[core]:x64-uwp...
-- Using cached C:/src/vcpkg/downloads/glennrp-libpng-v1.6.37.tar.gz
-- Cleaning sources at C:/src/vcpkg/buildtrees/libpng/src/v1.6.37-c993153cdf.clean. Pass --editable to vcpkg to reuse sources.
-- Extracting source C:/src/vcpkg/downloads/glennrp-libpng-v1.6.37.tar.gz
-- Applying patch use-abort-on-all-platforms.patch
-- Using source at C:/src/vcpkg/buildtrees/libpng/src/v1.6.37-c993153cdf.clean
-- Configuring x64-uwp
-- Building x64-uwp-dbg
-- Building x64-uwp-rel
-- Installing: C:/src/vcpkg/packages/libpng_x64-uwp/share/libpng/copyright
-- Performing post-build validation
-- Performing post-build validation done
Building package libpng[core]:x64-uwp... done
Installing package libpng[core]:x64-uwp...
Installing package libpng[core]:x64-uwp... done
Elapsed time for package libpng:x64-uwp: 15.31 s
Total elapsed time: 15.35 s
The package libpng:x64-uwp provides CMake targets:
find_package(libpng CONFIG REQUIRED)
target_link_libraries(main PRIVATE png)
```

Finally, to fully commit and publish the changes, we need to bump the internal release number and add the patch file to source control, then make a Pull Request!

```no-highlight
# ports\libpng\CONTROL
Source: libpng
Version: 1.6.24-1
Version: 1.6.37-1
Build-Depends: zlib
```
9 changes: 8 additions & 1 deletion scripts/cmake/vcpkg_extract_source_archive_ex.cmake
Original file line number Diff line number Diff line change
@@ -101,9 +101,16 @@ function(vcpkg_extract_source_archive_ex)
string(SHA512 PATCHSET_HASH ${PATCHSET_HASH})
string(SUBSTRING ${PATCHSET_HASH} 0 10 PATCHSET_HASH)
set(SOURCE_PATH "${_vesae_WORKING_DIRECTORY}/${SHORTENED_SANITIZED_REF}-${PATCHSET_HASH}")
if (NOT _VCPKG_EDITABLE)
string(APPEND SOURCE_PATH ".clean")
if(EXISTS ${SOURCE_PATH})
message(STATUS "Cleaning sources at ${SOURCE_PATH}. Pass --editable to vcpkg to reuse sources.")
file(REMOVE_RECURSE ${SOURCE_PATH})
endif()
endif()

if(NOT EXISTS ${SOURCE_PATH})
set(TEMP_DIR "${_vesae_WORKING_DIRECTORY}/TEMP")
set(TEMP_DIR "${_vesae_WORKING_DIRECTORY}/${SHORTENED_SANITIZED_REF}-${PATCHSET_HASH}.tmp")
file(REMOVE_RECURSE ${TEMP_DIR})
vcpkg_extract_source_archive("${_vesae_ARCHIVE}" "${TEMP_DIR}")

Loading

0 comments on commit f9bdf13

Please sign in to comment.