[question] Problem when uploading sources

[markusbahn]

What is your question?

Hi! To build our 3rd party dependencies in our CI/CD pipeline, we need to backup all the sources of our dependencies. I am currently testing the feature (https://docs.conan.io/2/devops/backup_sources/sources_backup.html) with the following minimal conanfile:

from conan import ConanFile
from conan.tools.cmake import cmake_layout


class TestRecipe(ConanFile):
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeToolchain", "CMakeDeps"

    def requirements(self):
        self.requires("zlib/1.3")

    def layout(self):
        cmake_layout(self)

conan install . --build=missing -r conan-center-remote gives me:

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu20
compiler.libcxx=libstdc++11
compiler.version=10
os=Linux

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu20
compiler.libcxx=libstdc++11
compiler.version=10
os=Linux


======== Computing dependency graph ========
zlib/1.3: Not found in local cache, looking in remotes...
zlib/1.3: Checking remote: conan-center-remote
zlib/1.3: Downloaded recipe revision 5c0f3a1a222eebb6bff34980bcd3e024
Graph root
    conanfile.py: /home/mzinser/test_conan/conanfile.py
Requirements
    zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024 - Downloaded (conan-center-remote)

======== Computing necessary packages ========
Requirements
    zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024:3f0cbc1394cb3fd9f3473d67001da1a496fb5b26 - Build

======== Installing packages ========
zlib/1.3: Sources downloaded from 'conan-center-remote'
zlib/1.3: Calling source() in /home/mzinser/.conan2/p/zlib7558f4f9c22fe/s/src

-------- Installing package zlib/1.3 (1 of 1) --------
zlib/1.3: Building from source
zlib/1.3: Package zlib/1.3:3f0cbc1394cb3fd9f3473d67001da1a496fb5b26
zlib/1.3: Copying sources to build folder
zlib/1.3: Building your package in /home/mzinser/.conan2/p/b/zlib560c9031eafb6/b
zlib/1.3: Calling generate()
zlib/1.3: Generators folder: /home/mzinser/.conan2/p/b/zlib560c9031eafb6/b/build/Release/generators
zlib/1.3: CMakeToolchain generated: conan_toolchain.cmake
zlib/1.3: CMakeToolchain generated: CMakePresets.json
zlib/1.3: CMakeToolchain generated: ../../../src/CMakeUserPresets.json
zlib/1.3: Generating aggregated env files
zlib/1.3: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
zlib/1.3: Calling build()
zlib/1.3: Apply patch (conan): separate static/shared builds, disable debug suffix, disable building examples
zlib/1.3: Running CMake.configure()
zlib/1.3: RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="/home/mzinser/.conan2/p/b/zlib560c9031eafb6/b/build/Release/generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/home/mzinser/.conan2/p/b/zlib560c9031eafb6/p" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/home/mzinser/.conan2/p/b/zlib560c9031eafb6/b/src"
-- Using Conan toolchain: /home/mzinser/.conan2/p/b/zlib560c9031eafb6/b/build/Release/generators/conan_toolchain.cmake
-- Conan toolchain: Setting CMAKE_POSITION_INDEPENDENT_CODE=ON (options.fPIC)
-- Conan toolchain: Setting BUILD_SHARED_LIBS = OFF
-- The C compiler identification is GNU 10.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - done
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Renaming
--     /home/mzinser/.conan2/p/b/zlib560c9031eafb6/b/src/zconf.h
-- to 'zconf.h.included' because this file is included with zlib
-- but CMake generates it automatically in the build directory.
-- Configuring done (1.6s)
-- Generating done (0.0s)
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_POLICY_DEFAULT_CMP0091


-- Build files have been written to: /home/mzinser/.conan2/p/b/zlib560c9031eafb6/b/build/Release

zlib/1.3: Running CMake.build()
zlib/1.3: RUN: cmake --build "/home/mzinser/.conan2/p/b/zlib560c9031eafb6/b/build/Release" -- -j8
[  6%] Building C object CMakeFiles/zlib.dir/adler32.c.o
[ 12%] Building C object CMakeFiles/zlib.dir/compress.c.o
[ 18%] Building C object CMakeFiles/zlib.dir/crc32.c.o
[ 25%] Building C object CMakeFiles/zlib.dir/deflate.c.o
[ 31%] Building C object CMakeFiles/zlib.dir/gzclose.c.o
[ 37%] Building C object CMakeFiles/zlib.dir/gzlib.c.o
[ 43%] Building C object CMakeFiles/zlib.dir/gzread.c.o
[ 50%] Building C object CMakeFiles/zlib.dir/gzwrite.c.o
[ 56%] Building C object CMakeFiles/zlib.dir/inflate.c.o
[ 62%] Building C object CMakeFiles/zlib.dir/infback.c.o
[ 68%] Building C object CMakeFiles/zlib.dir/inftrees.c.o
[ 75%] Building C object CMakeFiles/zlib.dir/inffast.c.o
[ 81%] Building C object CMakeFiles/zlib.dir/trees.c.o
[ 87%] Building C object CMakeFiles/zlib.dir/uncompr.c.o
[ 93%] Building C object CMakeFiles/zlib.dir/zutil.c.o
[100%] Linking C static library libz.a
[100%] Built target zlib

zlib/1.3: Package '3f0cbc1394cb3fd9f3473d67001da1a496fb5b26' built
zlib/1.3: Build folder /home/mzinser/.conan2/p/b/zlib560c9031eafb6/b/build/Release
zlib/1.3: Generating the package
zlib/1.3: Packaging in folder /home/mzinser/.conan2/p/b/zlib560c9031eafb6/p
zlib/1.3: Calling package()
zlib/1.3: Running CMake.install()
zlib/1.3: RUN: cmake --install "/home/mzinser/.conan2/p/b/zlib560c9031eafb6/b/build/Release" --prefix "/home/mzinser/.conan2/p/b/zlib560c9031eafb6/p"
-- Install configuration: "Release"
-- Installing: /home/mzinser/.conan2/p/b/zlib560c9031eafb6/p/lib/libz.a
-- Installing: /home/mzinser/.conan2/p/b/zlib560c9031eafb6/p/include/zconf.h
-- Installing: /home/mzinser/.conan2/p/b/zlib560c9031eafb6/p/include/zlib.h

zlib/1.3: package(): Packaged 2 '.h' files: zlib.h, zconf.h
zlib/1.3: package(): Packaged 1 '.a' file: libz.a
zlib/1.3: package(): Packaged 1 file: LICENSE
zlib/1.3: Created package revision 270bfd3ae0ba771458d3bd841a256e40
zlib/1.3: Package '3f0cbc1394cb3fd9f3473d67001da1a496fb5b26' created
zlib/1.3: Full package reference: zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024:3f0cbc1394cb3fd9f3473d67001da1a496fb5b26#270bfd3ae0ba771458d3bd841a256e40
zlib/1.3: Package folder /home/mzinser/.conan2/p/b/zlib560c9031eafb6/p
WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN: deprecated:     'cpp_info.names' used in: zlib/1.3

======== Finalizing install (deploy, generators) ========
conanfile.py: Writing generators to /home/mzinser/test_conan/build/Release/generators
conanfile.py: Generator 'CMakeDeps' calling 'generate()'
conanfile.py: Generator 'CMakeToolchain' calling 'generate()'
conanfile.py: CMakeToolchain generated: conan_toolchain.cmake
conanfile.py: Preset 'conan-release' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-release' if using CMake>=3.23
conanfile.py: If your CMake version is not compatible with CMakePresets (<3.23) call cmake like: 'cmake <path> -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=/home/mzinser/test_conan/build/Release/generators/conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Release'
conanfile.py: CMakeToolchain generated: CMakePresets.json
conanfile.py: Generating aggregated env files
conanfile.py: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished successfully

I've specified in my global.conf the core.sources:upload_url (a generic artifactory repository). When perform conan upload zlib/1.3, then I need to specify a remote. When I add a remote, I get the following output:

conan upload zlib/1.3 -r debug-conan-stage-dev-local

-------- Checking server existing packages --------
zlib/1.3: Checking which revisions exist in the remote server

-------- Preparing artifacts for upload --------
zlib/1.3:3f0cbc1394cb3fd9f3473d67001da1a496fb5b26: Compressing conan_package.tgz

-------- Uploading artifacts --------
Uploading recipe 'zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024'
Uploading package 'zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024:3f0cbc1394cb3fd9f3473d67001da1a496fb5b26#270bfd3ae0ba771458d3bd841a256e40'
Upload complete


-------- Upload summary --------
debug-conan-stage-dev-local
  zlib/1.3
    revisions
      5c0f3a1a222eebb6bff34980bcd3e024 (Uploaded)
        packages
          3f0cbc1394cb3fd9f3473d67001da1a496fb5b26
            revisions
              270bfd3ae0ba771458d3bd841a256e40 (Uploaded)

But no sources are uploaded to my generic repository. conan cache backup-upload zlib/1.3 -c -r debug-conan-stage-dev-local gives me no output and also no uploaded files.

What am I doing wrong?

Best regards,
Markus

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[AbrilRBS]

Hi @markusbahn thanks a lot for your question.

The logic for the backup sources only triggers if either core.sources:download_cache or core.sources:download_urls are explicitly set. Re-reading the documentation, this is not clearly explained, so we will look into improving it

Let me know if this helps and solves your issue, happy to help if not :)

[markusbahn]

Hi @AbrilRBS,

thanks for the help! Now it works. I also had to delete the .conan2 folder to trigger the download again. Otherwise conan would use the cached result.

[AbrilRBS]

Reopening so we don't forget to update the docs to better reflect this :)