git and crates.io behave differently

[folkertdev]

this is a weird one. I see different behavior between these dependency declarations

libz-sys = { git = "https://github.com/rust-lang/libz-sys.git", rev = "414ca0a22b35449ae297fb0775fad578d5c9a87a", default-features = false, features = ["zlib-ng"] } # use libz-ng in libz compat mode
libz-sys = { version = "1.1.18", default-features = false, features = ["zlib-ng"] } # use libz-ng in libz compat mode

that really is the commit of the 1.1.18 tag: https://github.com/rust-lang/libz-sys/releases/tag/1.1.18, and I expect those two commands to be equivalent. Yet, one fails and the other succeeds. When I look with -vvv, they really do quite different things.

The problem occurs when trying to cross-compile for the s390x-unknown-linux-gnu target, see cmake output below.

Logs

error: failed to run custom build command for libz-sys v1.1.18
Caused by:
process didn't exit successfully: /home/runner/work/zlib-rs/zlib-rs/target/debug/build/libz-sys-62ddb766a49e0499/build-script-build (exit status: 101)
--- stdout
cargo:rerun-if-env-changed=LIBZ_SYS_STATIC
cargo:rerun-if-changed=build.rs
cargo:rerun-if-changed=zng/cmake.rs
cargo:rerun-if-changed=zng/cc.rs
CMAKE_TOOLCHAIN_FILE_s390x-unknown-linux-gnu = None
CMAKE_TOOLCHAIN_FILE_s390x_unknown_linux_gnu = None
TARGET_CMAKE_TOOLCHAIN_FILE = None
CMAKE_TOOLCHAIN_FILE = None
CMAKE_GENERATOR_s390x-unknown-linux-gnu = None
CMAKE_GENERATOR_s390x_unknown_linux_gnu = None
TARGET_CMAKE_GENERATOR = None
CMAKE_GENERATOR = None
CMAKE_PREFIX_PATH_s390x-unknown-linux-gnu = None
CMAKE_PREFIX_PATH_s390x_unknown_linux_gnu = None
TARGET_CMAKE_PREFIX_PATH = None
CMAKE_PREFIX_PATH = None
CMAKE_s390x-unknown-linux-gnu = None
CMAKE_s390x_unknown_linux_gnu = None
TARGET_CMAKE = None
CMAKE = None
running: cd "/home/runner/work/zlib-rs/zlib-rs/target/s390x-unknown-linux-gnu/debug/build/libz-sys-337d39c9c58d4b39/out/build" && CMAKE_PREFIX_PATH="" "cmake" "/home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libz-sys-1.1.18/src/zlib-ng" "-DBUILD_SHARED_LIBS=OFF" "-DZLIB_COMPAT=ON" "-DZLIB_ENABLE_TESTS=OFF" "-DWITH_GZFILEOP=ON" "-DWITH_DFLTCC_DEFLATE=1" "-DWITH_DFLTCC_INFLATE=1" "-DCMAKE_SYSTEM_NAME=Linux" "-DCMAKE_SYSTEM_PROCESSOR=s390x" "-DCMAKE_INSTALL_PREFIX=/home/runner/work/zlib-rs/zlib-rs/target/s390x-unknown-linux-gnu/debug/build/libz-sys-337d39c9c58d4b39/out" "-DCMAKE_C_FLAGS= -DDFLTCC_LEVEL_MASK=0x7e -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_C_COMPILER=/usr/bin/s390x-linux-gnu-gcc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_CXX_COMPILER=/usr/bin/s390x-linux-gnu-g++" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_ASM_COMPILER=/usr/bin/s390x-linux-gnu-gcc" "-DCMAKE_BUILD_TYPE=RelWithDebInfo"
-- Using CMake version 3.30.2
-- ZLIB_HEADER_VERSION: 1.3.0
-- ZLIBNG_HEADER_VERSION: 2.1.6
-- The C compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/s390x-linux-gnu-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Arch not recognized, falling back to cmake arch: 's390x'
-- Basearch of 's390x' has been detected as: 's360'
-- Performing Test FNO_LTO_AVAILABLE
-- Performing Test FNO_LTO_AVAILABLE - Success
-- Looking for arm_acle.h
-- Looking for arm_acle.h - not found
-- Looking for sys/auxv.h
-- Looking for sys/auxv.h - found
-- Looking for sys/sdt.h
-- Looking for sys/sdt.h - not found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - done
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for strerror
-- Looking for strerror - found
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Looking for aligned_alloc
-- Looking for aligned_alloc - found
-- Performing Test HAVE_NO_INTERPOSITION
-- Performing Test HAVE_NO_INTERPOSITION - Success
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN - Success
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL - Success
-- Performing Test HAVE_ATTRIBUTE_ALIGNED
-- Performing Test HAVE_ATTRIBUTE_ALIGNED - Success
-- Performing Test HAVE_BUILTIN_CTZ
-- Performing Test HAVE_BUILTIN_CTZ - Success
-- Performing Test HAVE_BUILTIN_CTZLL
-- Performing Test HAVE_BUILTIN_CTZLL - Success
-- Performing Test HAVE_PTRDIFF_T
-- Performing Test HAVE_PTRDIFF_T - Success
-- Performing Test HAVE_S390_INTRIN
-- Performing Test HAVE_S390_INTRIN - Success
-- Performing Test HAVE_VGFMA_INTRIN
-- Performing Test HAVE_VGFMA_INTRIN - Success
-- Architecture-specific source files: arch/s390/s390_features.c;arch/s390/dfltcc_common.c;arch/s390/dfltcc_deflate.c;arch/s390/dfltcc_inflate.c;arch/s390/crc32-vx.c
-- The following features have been enabled:

• CMAKE_BUILD_TYPE, Build type: RelWithDebInfo (selected)
• WITH_GZFILEOP, Compile with support for gzFile related functions
• ZLIB_COMPAT, Compile with zlib compatible API
• ZLIBNG_ENABLE_TESTS, Test zlib-ng specific API
• WITH_SANITIZER, Enable sanitizer support
• WITH_GTEST, Build gtest_zlib
• WITH_OPTIM, Build with optimisation
• WITH_NEW_STRATEGIES, Use new strategies
• WITH_DFLTCC_DEFLATE, Build with DFLTCC intrinsics for compression on IBM Z
• WITH_DFLTCC_INFLATE, Build with DFLTCC intrinsics for decompression on IBM Z
• WITH_CRC32_VX, Build with vectorized CRC32 on IBM Z
  -- The following features have been disabled:
• ZLIB_SYMBOL_PREFIX, Publicly exported symbols DO NOT have a custom prefix
• ZLIB_ENABLE_TESTS, Build test binaries
• WITH_FUZZERS, Build test/fuzz
• WITH_BENCHMARKS, Build test/benchmarks
• WITH_BENCHMARK_APPS, Build application benchmarks
• WITH_NATIVE_INSTRUCTIONS, Instruct the compiler to use the full instruction set on this host (gcc/clang -march=native)
• WITH_MAINTAINER_WARNINGS, Build with project maintainer warnings
• WITH_CODE_COVERAGE, Enable code coverage reporting
• WITH_INFLATE_STRICT, Build with strict inflate distance checking
• WITH_INFLATE_ALLOW_INVALID_DIST, Build with zero fill for inflate invalid distances
• INSTALL_UTILS, Copy minigzip and minideflate during install
  -- Configuring done (3.0s)
  --- stderr
  CMake Error at CMakeLists.txt:1059 (add_library):
  Cannot find source file:
  arch/s390/s390_features.h
  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm
  .ccm .cxxm .c++m .h .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90
  .f95 .f03 .hip .ispc
  CMake Error at CMakeLists.txt:1059 (add_library):
  No SOURCES given to target: zlib
  CMake Generate step failed. Build files cannot be regenerated correctly.
  thread 'main' panicked at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cmake-0.1.50/src/lib.rs:1098:5:
  command did not execute successfully, got: exit status: 1
  build script failed, must exit now
  stack backtrace:
  0: rust_begin_unwind
  at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:652:5
  1: core::panicking::panic_fmt
  at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/panicking.rs:72:14
  2: cmake::fail
  at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cmake-0.1.50/src/lib.rs:1098:5
  3: cmake::run
  at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cmake-0.1.50/src/lib.rs:1076:9
  4: cmake::Config::build
  at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cmake-0.1.50/src/lib.rs:813:13
  5: build_script_build::zng::build_zng::build_zlib_ng
  at ./zng/cmake.rs:21:23
  6: build_script_build::build_zlib_ng
  at ./build.rs:201:5
  7: build_script_build::main
  at ./build.rs:22:16
  8: core::ops::function::FnOnce::call_once
  at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/ops/function.rs:250:5
  note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
  warning: build failed, waiting for other jobs to finish...
  error: command /home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo test --no-run --message-format json-render-diagnostics --target s390x-unknown-linux-gnu exited with code 101
  Error: Process completed with exit code 101.

So, wth? is the tag different in practice from the release on crates.io? That would be bad, I think? Other crates.io versions also have this problem though.

with cargo -vvv you can actually see the cmake logs, and they really are very different!

https://gist.github.com/folkertdev/0b0dcb12f934d318a1dd1a9deaae8376

[Byron]

That is very interesting indeed! From all I can tell, after running cargo publish successfully a tag is created, so the versions would match.
Since other crates.io versions have this problem as well, it seems to be a systemic issue, rather than an exceptional one.

Something I wonder is if the crate libz-ng-sys has similar issues - it's only available on crates.io but should be the same as libz-ng with the zlib-ng feature enabled.

In any case, maybe the reason for this is in the build script, it's quite complex by now.

[folkertdev]

I made a better diff https://gist.github.com/folkertdev/0b0dcb12f934d318a1dd1a9deaae8376#file-difference-diff, and actually now it looks like the commands are quite similar. (a diff viewer with line wrapping is the easiest way to see that the only real differences are in hashes and git/checkout/... instead of registry/...).

So, then it could be some sort of environment variable? It's difficult to debug of course because the failing version is the crates.io version.

libz-ng-sys does not seem to support cross-compilation at the moment?

  = note: /usr/bin/ld: /home/folkertdev/rust/crates-io-bug/target/s390x-unknown-linux-gnu/debug/deps/crates_io_bug-44e3d326b4011c8b.2uuh1lvi20jyf749j504s17j0.rcgu.o: Relocations in generic ELF (EM: 22)
          /usr/bin/ld: /home/folkertdev/rust/crates-io-bug/target/s390x-unknown-linux-gnu/debug/deps/crates_io_bug-44e3d326b4011c8b.2uuh1lvi20jyf749j504s17j0.rcgu.o: Relocations in generic ELF (EM: 22)
          /usr/bin/ld: /home/folkertdev/rust/crates-io-bug/target/s390x-unknown-linux-gnu/debug/deps/crates_io_bug-44e3d326b4011c8b.2uuh1lvi20jyf749j504s17j0.rcgu.o: Relocations in generic ELF (EM: 22)
          /usr/bin/ld: /home/folkertdev/rust/crates-io-bug/target/s390x-unknown-linux-gnu/debug/deps/crates_io_bug-44e3d326b4011c8b.2uuh1lvi20jyf749j504s17j0.rcgu.o: Relocations in generic ELF (EM: 22)
          /usr/bin/ld: /home/folkertdev/rust/crates-io-bug/target/s390x-unknown-linux-gnu/debug/deps/crates_io_bug-44e3d326b4011c8b.2uuh1lvi20jyf749j504s17j0.rcgu.o: Relocations in generic ELF (EM: 22)
          /usr/bin/ld: /home/folkertdev/rust/crates-io-bug/target/s390x-unknown-linux-gnu/debug/deps/crates_io_bug-44e3d326b4011c8b.2uuh1lvi20jyf749j504s17j0.rcgu.o: error adding symbols: file in wrong format

so that's a separate thing to fix, given that it sort of should run the same logic as libz-sys

[folkertdev]

edit to the above: apparently even for cargo build the runner must be configured. Not sure why that is, TIL.

So then, libz-ng-sys does not have the problem that I run into with libz-sys, which is an interesting clue, which made me look at the files that actually get, and here's our problem

folkertdev@folkertdev:~/rust/libz-sys$ ls /home/folkertdev/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libz-sys-1.1.18/src/zlib-ng/arch 
arm  generic  power  riscv  x86
folkertdev@folkertdev:~/rust/libz-sys$ ls /home/folkertdev/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libz-ng-sys-1.1.15/src/zlib-ng/arch 
arm  generic  power  riscv  s390  x86

the whole s390 directory really just isn't there in the release of libz-sys. That is because in version 1.1.18, the Cargo.toml has this list

include = [
    "LICENSE*",
    "/README.md",
    "build.rs",
    "zng/cc.rs",
    "zng/cmake.rs",
    "src/*.rs",
    "src/*.c",
    "src/zlib/*.[ch]",
    "src/zlib/*.pc.in",
    # zlib-ng
    "src/zlib-ng/**.[ch]",
    "src/zlib-ng/arch/arm/**.[ch]",
    "src/zlib-ng/arch/generic/**.[ch]",
    "src/zlib-ng/arch/power/**.[ch]",
    "src/zlib-ng/arch/riscv/**.[ch]",
    "src/zlib-ng/arch/s390x/**.[ch]",
    "src/zlib-ng/arch/x86/**.[ch]",
    "src/zlib-ng/*.[ch].in",
    "src/zlib-ng/*.pc.in",
    "src/zlib-ng/zlib_name_mangling.h.empty",
    # zlib-ng cmake
    "src/zlib-ng/CMakeLists.txt",
    "src/zlib-ng/zlib.pc.cmakein",
    "src/zlib-ng/cmake",
]

where s390x is written instead of the s390 that zlib-ng uses (not sure which is actually correct, the target uses the x, but whatever, for inclusion libz-sys should follow zlib-ng here).

Now it appears that on main, an exclude = list is used instead, so I think the fix for me specifically would be a 1.1.19 release. That should hopefully include the file.

My theory is that libz-ng-sys is old enough that that folder was not renamed, or maybe it retains the files for some other reason.