Package Acquisition using Source / CMake Fetch Content

[ronniegeraghty]

Currently, directly including the source for 3rd party dependencies is the most common why C++ developers integrate dependencies into their projects. The Azure SDK for C++ is supposed to support this through CMake Fetch Content, allowing it to take care of downloading the source code at a specific version and linking the build process for the specific Azure SDK library. But it seems our support for CMake Fetch Content has lapsed.

Current CMake Fetch Content Issue:

Project Set Up

• main.cpp file:

#include <azure/identity.hpp>
#include <azure/keyvault/secrets.hpp>
#include <iostream>

using namespace Azure::Security::KeyVault::Secrets;

int main()
{
    try
    {
        // Set Key Vault URL string
        auto const keyVaultUrl = std::getenv("AZURE_KEYVAULT_URL");

        // Create Default Azure Credential to Authenticate.
        // It will pick up on our AzureCLI login
        auto credential = std::make_shared<Azure::Identity::DefaultAzureCredential>();

        // Create Key Vault Secret Client
        SecretClient secretClient(keyVaultUrl, credential);

        // Create a Secret
        std::string secretName("MySampleSecret");
        std::string secretValue("My super secret value");
        secretClient.SetSecret(secretName, secretValue);

        // Get the Secret
        KeyVaultSecret secret = secretClient.GetSecret(secretName).Value;
        std::string valueString = secret.Value.HasValue()
                                      ? secret.Value.Value()
                                      : "NONE RETURNED";
        std::cout << "Secret is returned with name " << secret.Name
                  << " and value " << valueString << std::endl;
    }
    catch (Azure::Core::Credentials::AuthenticationException const &e)
    {
        std::cout << "Authentication Exception happened:" << std::endl
                  << e.what() << std::endl;
        return 1;
    }
    catch (Azure::Core::RequestFailedException const &e)
    {
        std::cout << "Key Vault Secret Client Exception happened:" << std::endl
                  << e.Message << std::endl;
        return 1;
    }

    return 0;
}

• CMakeLists.txt file

# Specify the minimum version of CMake required to build this project
cmake_minimum_required(VERSION 3.30.0)

# Define the project name, version, and the languages used
project(azure_sample VERSION 0.1.0 LANGUAGES C CXX)

# Fetch content
include(FetchContent)
FetchContent_Declare(azuresdkforcpp
    # Set the SDK URL path and releasse tag
    GIT_REPOSITORY      https://github.com/Azure/azure-sdk-for-cpp.git
    GIT_TAG             azure-security-keyvault-secrets_4.2.1
)
FetchContent_GetProperties(azuresdkforcpp)
if(NOT azuresdkforcpp_POPULATED)
    FetchContent_MakeAvailable(azuresdkforcpp)
    add_subdirectory(${azuresdkforcpp_SOURCE_DIR} ${azuresdkforcpp_BINARY_DIR} EXCLUDE_FROM_ALL)
endif()

add_executable(azure_sample main.cpp)

target_compile_definitions(azure_sample PRIVATE _azrue_BUILD_SAMPLES)

target_link_libraries(azure_sample Azure::azure-security-keyvault-secrets)

Windows Result

On Windows, I hit an issue running the CMake config command: cmake -B ./build

PS C:\Users\user\CMakeFetchContentSample> cmake -B ./build
-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.26100.0 to target Windows 10.0.22631.
-- The C compiler identification is MSVC 19.42.34436.0
-- The CXX compiler identification is MSVC 19.42.34436.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/IntPreview/VC/Tools/MSVC/14.42.34433/bin/HostX64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/IntPreview/VC/Tools/MSVC/14.42.34433/bin/HostX64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Cloning into 'azuresdkforcpp-src'...
HEAD is now at 1d22b9bc5 Fix broken link (#5273)
No transport adapter was selected, using WinHTTP as the default option for Windows.
Vcpkg integrate step.
CMAKE_TOOLCHAIN_FILE is not defined. Define it for the user.
AZURE_SDK_DISABLE_AUTO_VCPKG is not defined. Fetch a local copy of vcpkg.
Vcpkg commit string used: 43cf47eccfbe27006cf9534a5db809798f8c37fe
CMake Warning (dev) at C:/Program Files/CMake/share/cmake-3.31/Modules/FetchContent.cmake:1953 (message):
  Calling FetchContent_Populate(vcpkg) is deprecated, call
  FetchContent_MakeAvailable(vcpkg) instead.  Policy CMP0169 can be set to
  OLD to allow FetchContent_Populate(vcpkg) to be called directly for now,
  but the ability to call it with declared details will be removed completely
  in a future version.
Call Stack (most recent call first):
  build/_deps/azuresdkforcpp-src/cmake-modules/AzureVcpkg.cmake:36 (FetchContent_Populate)
  build/_deps/azuresdkforcpp-src/CMakeLists.txt:25 (az_vcpkg_integrate)
This warning is for project developers.  Use -Wno-dev to suppress it.

Vcpkg integrate step - DONE.
-- CMAKE_SYSTEM_NAME=Windows;
-- VCPKG_TARGET_TRIPLET=;
Vcpkg integrate step.
Vcpkg integrate step - DONE.
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Found Threads: TRUE
CMake Error at build/_deps/azuresdkforcpp-src/sdk/core/azure-core/CMakeLists.txt:37 (find_package):
  Could not find a package configuration file provided by "wil" with any of
  the following names:

    wilConfig.cmake
    wil-config.cmake

  Add the installation prefix of "wil" to CMAKE_PREFIX_PATH or set "wil_DIR"
  to a directory containing one of the above files.  If "wil" provides a
  separate development package or SDK, be sure it has been installed.


-- Configuring incomplete, errors occurred!

Our current build process using CMake Fetch Content doesn't seem to either be properly pulling the wil dependency or is not properly able to locate the dependency.

Linux Result

On Linux (WSL/Ubuntu) I also hit an issue running the CMake config command: cmake -B ./build:

ronnieg@---:~/code/cmakefetchcontentsample$ cmake -B ./build
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Cloning into 'azuresdkforcpp-src'...
HEAD is now at 1d22b9bc5 Fix broken link (#5273)
No transport adapter was selected, using libcurl as the default option for POSIX.
Vcpkg integrate step.
CMAKE_TOOLCHAIN_FILE is not defined. Define it for the user.
AZURE_SDK_DISABLE_AUTO_VCPKG is not defined. Fetch a local copy of vcpkg.
Vcpkg commit string used: 43cf47eccfbe27006cf9534a5db809798f8c37fe
CMake Warning (dev) at /home/ronnieg/cmake-3.31.2-linux-x86_64/share/cmake-3.31/Modules/FetchContent.cmake:1953 (message):
  Calling FetchContent_Populate(vcpkg) is deprecated, call
  FetchContent_MakeAvailable(vcpkg) instead.  Policy CMP0169 can be set to
  OLD to allow FetchContent_Populate(vcpkg) to be called directly for now,
  but the ability to call it with declared details will be removed completely
  in a future version.
Call Stack (most recent call first):
  build/_deps/azuresdkforcpp-src/cmake-modules/AzureVcpkg.cmake:36 (FetchContent_Populate)
  build/_deps/azuresdkforcpp-src/CMakeLists.txt:25 (az_vcpkg_integrate)
This warning is for project developers.  Use -Wno-dev to suppress it.

Vcpkg integrate step - DONE.
-- CMAKE_SYSTEM_NAME=Linux;
-- VCPKG_TARGET_TRIPLET=;
Vcpkg integrate step.
Vcpkg integrate step - DONE.
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
CMake Error at /home/ronnieg/cmake-3.31.2-linux-x86_64/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find CURL (missing: CURL_LIBRARY CURL_INCLUDE_DIR)
Call Stack (most recent call first):
  /home/ronnieg/cmake-3.31.2-linux-x86_64/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)
  /home/ronnieg/cmake-3.31.2-linux-x86_64/share/cmake-3.31/Modules/FindCURL.cmake:203 (find_package_handle_standard_args)
  build/_deps/azuresdkforcpp-src/sdk/core/azure-core/CMakeLists.txt:31 (find_package)


-- Configuring incomplete, errors occurred!

Seems like this is just a case of missing curl. I installed curl and cleared the CMake cache and tried again. The next dependency I was told was missing was OpenSSL, so again I installed it and reran the config. Then I ran into the following issue:

ronnieg@CPC-rgera-JR9LR:~/code/cmakefetchcontentsample$ cmake -B ./build
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Cloning into 'azuresdkforcpp-src'...
HEAD is now at 1d22b9bc5 Fix broken link (#5273)
No transport adapter was selected, using libcurl as the default option for POSIX.
Vcpkg integrate step.
CMAKE_TOOLCHAIN_FILE is not defined. Define it for the user.
AZURE_SDK_DISABLE_AUTO_VCPKG is not defined. Fetch a local copy of vcpkg.
Vcpkg commit string used: 43cf47eccfbe27006cf9534a5db809798f8c37fe
CMake Warning (dev) at /home/ronnieg/cmake-3.31.2-linux-x86_64/share/cmake-3.31/Modules/FetchContent.cmake:1953 (message):
  Calling FetchContent_Populate(vcpkg) is deprecated, call
  FetchContent_MakeAvailable(vcpkg) instead.  Policy CMP0169 can be set to
  OLD to allow FetchContent_Populate(vcpkg) to be called directly for now,
  but the ability to call it with declared details will be removed completely
  in a future version.
Call Stack (most recent call first):
  build/_deps/azuresdkforcpp-src/cmake-modules/AzureVcpkg.cmake:36 (FetchContent_Populate)
  build/_deps/azuresdkforcpp-src/CMakeLists.txt:25 (az_vcpkg_integrate)
This warning is for project developers.  Use -Wno-dev to suppress it.

Vcpkg integrate step - DONE.
-- CMAKE_SYSTEM_NAME=Linux;
-- VCPKG_TARGET_TRIPLET=;
Vcpkg integrate step.
Vcpkg integrate step - DONE.
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found CURL: /usr/lib/x86_64-linux-gnu/libcurl.so (found version "8.5.0")
Libcurl version 8.5.0
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "3.0.13")
-- Retrieving version from /home/ronnieg/code/cmakefetchcontentsample/build/_deps/azuresdkforcpp-src/sdk/core/azure-core/src/private/package_version.hpp
-- VERSION_MAJOR 1
-- VERSION_MINOR 12
-- VERSION_PATCH 0
-- VERSION_PRERELEASE beta.1
-- AZ_LIBRARY_VERSION
-- Including AMQP library
Vcpkg integrate step.
Vcpkg integrate step - DONE.
-- Using vendored uamqp
CMake Deprecation Warning at build/_deps/azuresdkforcpp-src/sdk/core/azure-core-amqp/vendor/azure-uamqp-c/CMakeLists.txt:4 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.


CMake Error at build/_deps/azuresdkforcpp-src/sdk/core/azure-core-amqp/vendor/azure-uamqp-c/CMakeLists.txt:56 (find_package):
  Could not find a package configuration file provided by
  "azure_macro_utils_c" with any of the following names:

    azure_macro_utils_cConfig.cmake
    azure_macro_utils_c-config.cmake

  Add the installation prefix of "azure_macro_utils_c" to CMAKE_PREFIX_PATH
  or set "azure_macro_utils_c_DIR" to a directory containing one of the above
  files.  If "azure_macro_utils_c" provides a separate development package or
  SDK, be sure it has been installed.


-- Configuring incomplete, errors occurred!

This new error appears to be caused by missing the azure_macro_utils_c dependency that is required by the SDK's vendored version of azure-uamqp-c. I understand that this could probably also by installing the dependency on my env and moving on, but I question why the build process is attempting to build any part of our AMQP stack when I'm only trying to use our Key Vault Secrets and Identity libraries.

Solutions for building the Azure SDK from source

• Fix current CMake Fetch Content scenario that invokes vcpkg underneath
• Update CMake Fetch Content scenario to only pull the desired source (maybe included core) and have the user resolve all external dependencies
• Users pull the zips from our releases in GitHub and they include only the desired library (maybe a compatible version of core as well) then users can either build that source for their platform and take a dep on the build binary or include the source in their project and build in their project's build

[ronniegeraghty]

Update

I attempted to set up a sample azure project using the Key Vault Secrets and Identity libraries when the dependencies are kept as source in the project. After tinkering around in the project's top level CMakeLists.txt file and the CMakeLists.txt files of the Secrets, Identity and Core libraries I was able to get something that builds and runs properly.
Link to sample project repo

All dependencies were place in the top level lib/ directory, including our cmake-modules/ directory as the CMakeLists.txt files for the libraries require them and I didn't want to edit those more than I had too.

The only changes I had to make to the CMakeLists.txt files of the azure service/core libraries were:

• alter the path where they look for the cmake-modules
• comment out all steps that would invoke vcpkg and cmake-modules related to vcpkg
• in the secrets and identity CMakeLists.txt file, I commented out the find_package(azure-core ....) lines, since my top level CMakeLists.txt file already added core as a subdirectory

Note

I only tested the sample on Linux (WSL/Ubuntu) as it made acquiring the dependencies for core easier.

Benefits

Over the current status of CMake Fetch Content, this solution allowed me to:

• only copy the source for libraries I require into my project (manually extracted and copied from a clone of the repo @ tag azure-security-keyvault-secrets_4.2.1
  • I understand this means the source for Identity I have may not be a stable source, but if that became an issue, I would have grabbed the Identity source from a tag related to one of its stable releases
• I did not run into issues like when using CMake Fetch Content, where it seems CMake tries to build all the repo, or at least all of the core libraries, so I didn't run into issues with azure-core-amqp needing azure_macro_utils_c.