Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AutoBuild] Split log files into a separate tarball #1229

Merged
merged 2 commits into from
Sep 19, 2022

Conversation

giordano
Copy link
Member

@giordano giordano commented Sep 18, 2022

Long term plan is still to do #778, but that's a lot of work, and we have little time at the moment. Simply splitting the logs into a separate tarball, instead, is a simpler short-term solution which has some very important bonuses:

  • the git tree hash of the main tarball should be reproducible (as long as you use the same toolchain)
  • which means we can start testing reproducibility of builds
  • rebuilding a package without changing its content won't create a new artifact: less strain on the Package Storage Server.

I still need to do tests about deployment on Yggdrasil though (Edit: see JuliaPackaging/Yggdrasil#5526).

Long term plan is still to do JuliaPackaging#778, but that's a lot of work, and we have little
time at the moment.  Simply splitting the logs into a separate tarball, instead,
is a simpler short-term solution which has some very important bonuses:

- the git tree hash of the main tarball should be reproducible (as long as you
use the same toolchain)
- which means we can start testing reproducibility of builds
- rebuilding a package without changing its content won't create a new artifact:
  less strain on the Package Storage Server.
@giordano
Copy link
Member Author

giordano commented Sep 18, 2022

Uhm, the failure

[ Info: Building for x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx29-julia_version+1.7.2, x86_64-unknown-freebsd
JLLs - building: Error During Test at /data/staticfloat/bb_azp/rootfs/agent_1/_work/1/s/test/jll.jl:41
  Got exception outside of a @test
  Unable to locate LibraryProduct(["libfoo"], :libfoo, String[], false, Symbol[]) within /data/staticfloat/bb_azp/rootfs/agent_1/tmp/jl_26NC7I for x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx29-julia_version+1.7.2
  Stacktrace:
    [1] error(s::String)
      @ Base ./error.jl:33
    [2] (::BinaryBuilder.var"#45#49"{Bool, Vector{Product}, String, Vector{String}, Dict{Any, Any}, Platform, String, String})(dest_prefix::String)
      @ BinaryBuilder /data/staticfloat/bb_azp/rootfs/agent_1/_work/1/s/src/AutoBuild.jl:1174
    [3] mktempdir(fn::BinaryBuilder.var"#45#49"{Bool, Vector{Product}, String, Vector{String}, Dict{Any, Any}, Platform, String, String}, parent::String; prefix::String)
      @ Base.Filesystem ./file.jl:750
    [4] mktempdir (repeats 2 times)
      @ ./file.jl:748 [inlined]
    [5] rebuild_jll_package(name::String, build_version::VersionNumber, sources::Vector{DirectorySource}, platforms::Vector{Platform}, products::Vector{Product}, dependencies::Vector{Dependency}, download_dir::String, upload_prefix::String; code_dir::String, verbose::Bool, from_scratch::Bool, kwargs::Base.Pairs{Symbol, Any, NTuple{4, Symbol}, NamedTuple{(:lazy_artifacts, :julia_compat, :init_block, :augment_platform_block), Tuple{Bool, String, String, String}}})
      @ BinaryBuilder /data/staticfloat/bb_azp/rootfs/agent_1/_work/1/s/src/AutoBuild.jl:1161
    [6] rebuild_jll_package(obj::Dict{String, Any}; download_dir::String, upload_prefix::String, build_version::Nothing, gh_org::String, verbose::Bool, from_scratch::Bool)
      @ BinaryBuilder /data/staticfloat/bb_azp/rootfs/agent_1/_work/1/s/src/AutoBuild.jl:1103
    [7] (::var"#134#138"{String, String, Platform})()
      @ Main /data/staticfloat/bb_azp/rootfs/agent_1/_work/1/s/test/jll.jl:160
    [8] withenv(::var"#134#138"{String, String, Platform}, ::Pair{String, String}, ::Vararg{Pair{String, String}})
      @ Base ./env.jl:172
    [9] (::var"#133#137")(build_path::String)
      @ Main /data/staticfloat/bb_azp/rootfs/agent_1/_work/1/s/test/jll.jl:103
   [10] mktempdir(fn::var"#133#137", parent::String; prefix::String)
      @ Base.Filesystem ./file.jl:750
   [11] mktempdir(fn::Function, parent::String) (repeats 2 times)
      @ Base.Filesystem ./file.jl:748
   [12] macro expansion
      @ /data/staticfloat/bb_azp/rootfs/agent_1/_work/1/s/test/jll.jl:42 [inlined]
   [13] macro expansion
      @ /data/staticfloat/bb_azp/rootfs/usr/local/julia-1.7/share/julia/stdlib/v1.7/Test/src/Test.jl:1283 [inlined]
   [14] top-level scope
      @ /data/staticfloat/bb_azp/rootfs/agent_1/_work/1/s/test/jll.jl:42
   [15] include(fname::String)
      @ Base.MainInclude ./client.jl:451
   [16] top-level scope
      @ /data/staticfloat/bb_azp/rootfs/agent_1/_work/1/s/test/runtests.jl:72
   [17] include(fname::String)
      @ Base.MainInclude ./client.jl:451
   [18] top-level scope
      @ none:6
   [19] eval
      @ ./boot.jl:373 [inlined]
   [20] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:268
   [21] _start()
      @ Base ./client.jl:495

looks unrelated to the changes to this package in this PR, but maybe some other upstream changes in BBB broke it (but we haven't seen any widespread breakage). I'll have to dig into this.

Edit: I did a first round of investigation and I found it's precisely the change to autobuild in this PR which breaks the test. It's weird, I can't quickly tell how this is related to that test, will have to dig more.

@giordano giordano marked this pull request as ready for review September 19, 2022 11:34
@giordano giordano merged commit c1a0d6a into JuliaPackaging:master Sep 19, 2022
@giordano giordano deleted the mg/split-tarballs branch September 19, 2022 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants