split libpolymake_julia and use artifacts

[benlorenz]

Since I don't really want to spend more time on making the old build-system nicer (e.g. support custom installation paths), this is an updated version of #304 using artifacts.
At the moment this comes with a fixed Manifest.toml to fetch the polymake_jll and libpolymake_julia_jll from my github repos, but it does already test successfully even on an old Ubuntu 16.04 that we couldn't support without libpolymake-julia separately.

libpolymake-julia lives here: https://github.com/oscar-system/libpolymake-julia
binaries: https://github.com/benlorenz/libpolymake_julia_jll.jl and https://github.com/benlorenz/polymake_jll.jl

It would be great if @kalmarek or @saschatimme can have a look, also at the builders, and especially the init_block.

I can give some details / explanations for the builders later.

advantages:

• this should fix any GLIBCXX errors
• removes the need for the xcode workaround
• Pkg artifacts, build.jl is gone !
• faster installation as no compilation is needed
• support for compilers supporting C++14 but not C++17 (e.g ubuntu 16.04)
• custom installations via Overrides.toml (for all Artifacts-enabled packages at once)

disadvantages:

• we need to drop support for cxxwrap 0.9

TODO:

• get builders into Yggdrasil
• I think we should store build-time libcxxwrap-julia version within libpolymake-julia for the check in src/setup_types.jl
• rebase
• add instructions for building libpolymake-julia and using Overrides.toml
• add warning when unsupported ENV["POLYMAKE_CONFIG"] is set

[fingolfin]

@benlorenz great! Perhaps turn JuliaPackaging/Yggdrasil@master...benlorenz:polymake into a draft PR? Then one could use the GitHub review features to comment on it (also, it doesn't hurt to see the result of the build tests there).

As to instructions for using Overrides.toml, of course want those for GAP.jl and (soon) also Singular.jl and Nemo.jl, so perhaps we can have some synergy here instead of writing more or less the same text over and over again. IIRC either @ThomasBreuer or @rbehrends (?) already used it successfully? If so, perhaps they'd be willing to write up a simple guide for whichever package they used it; then we can clean that up and share it with the others.

[ThomasBreuer]

Concerning the use of Overrides.toml:

Yes, I have tried this at the department in Aachen, in order to reduce the space/maintenance overhead for users, as follows.

• Assume that there is already a working installation of the packages in question for some user, with the artifacts in /users/oscar/.julia/artifacts, say.
• Before one asks Julia to install the packages in question for one's own account, one creates/extends the file ~/.julia/artifacts/Overrides.toml, such that it contains "redirections" of the form

848f0f318405266896d24d11e4baf5cf7bf3f203 = "/users/oscar/.julia/artifacts/848f0f318405266896d24d11e4baf5cf7bf3f203"

(for GAP),

d54a2909808f0f05123ef98f8eef4fcde1073e52 = "/users/oscar/.julia/artifacts/d54a2909808f0f05123ef98f8eef4fcde1073e52"

(for cxxwrap), etc.

• Then one starts Julia and calls using Pkg; Pkg.add("GAP"); Pkg.add("Polymake");, with the effect that the artifacts listed in Overrides.toml are not downloaded to ~/.julia/artifacts, but the ones from /users/oscar/.julia/artifacts are used.

This worked without problems.

If one wants to be more general, one should mention how to customize the path ~/.julia, and one can think about generating Overrides.toml automatically from a known path of the working installation.

Where would be an appropriate place for such instructions?

[fingolfin]

First off, we have to decide whether to put the instructions into a single place, and then link to that from everywhere, or whether we want to duplicate it in each involved package.

The former has the advantage of improvements and fixes only having to be made in a single place. Moreover, it could talk about how to do this for multiple packages at once etc.

The latter has the advantage that the instructions can be customized for the relevant package.

I think Overrides.toml is really something for powerusers (mostly: developers of the involved packages). So it doesn't need to be in the package manual; although I also don't mind if it's there. It could be in a dedicated README.devs.md file; or in a Wiki (every Github repository has a Wiki).

Anyway, lots of options... I suggest this: we start by putting the instructions into a page on the Oscar.jl wiki. There, it is easy for multiple of us to edit it until we like it. Once it is "good enough", we can still worry about duplicating it to each package or moving it into a file in the Oscar.jl repo, or ... But that decision shouldn't delay just writing the whole thing ;-).

CC @fieker in case he wants to add some more bike shed color options :-)

[benlorenz]

@benlorenz great! Perhaps turn JuliaPackaging/Yggdrasil@master...benlorenz:polymake into a draft PR? Then one could use the GitHub review features to comment on it (also, it doesn't hurt to see the result of the build tests there).

I wanted to do this before I went away for two weeks but was somewhat short of free time, I will try to do this until the beginning of August.

The instructions I was referring to (cmake build + override) should be put in the currently empty readme of the libpolymake-julia project. This can be kept rather short, is meant for developers but is needed for everyone who wants to use a custom polymake installation.
This should be independent of some more detailed and Oscar specific guide on the Oscar web-page.

[rbehrends]

I've written this simple script to find the artifacts and their hashes for a given Julia package (which must have been added with Pkg.add() or Pkg.develop()) before:

using Pkg, Pkg.Artifacts

for pkg in ARGS
  eval(Meta.parse("using $(pkg)"))
end

for pkg in ARGS
  println("Artifacts for $(pkg):")
  artifacts_toml = find_artifacts_toml(pathof(eval(Symbol(pkg))))
  if artifacts_toml === nothing
    println("  none")
  else
    artifacts = Pkg.Artifacts.load_artifacts_toml(artifacts_toml)
    for (artifact, meta) in artifacts
      println("  $(artifact) => $(meta["git-tree-sha1"])")
    end
  end
end

Example use:

$ julia artifacts.jl GAP
Artifacts for GAP:
  gap => 848f0f318405266896d24d11e4baf5cf7bf3f203

This gives you the hash to plug into Overrides.toml. Note that the directory structure for the override needs to be the same as for the tarball.

[benlorenz]

Draft for polymake_jll created: JuliaPackaging/Yggdrasil#1368

[benlorenz]

All todo items should be done:

• There is a minimal build-guide in the libpolymake-julia repository
• builders are in yggdrasil
• warning for polymake-config env var
• build-time libcxxwrap version is available as a function

So I think this is ready.

Probably we should add a short guide for a development workflow somewhere but this can be done later. Maybe someone else can experiment a little.

[kalmarek]

tearing those apart must have been a serious job! 🎉

[benlorenz]

tearing those apart must have been a serious job!

That went surprisingly quick, adjusting the cmake stuff was the most annoying part. Still a lot less work than getting perl and polymake to build and run with the artifacts.

One thing that remains is a github workflow file for the libpolymake-julia repo: It should build the library, add overrides and run some tests. Probably the Polymake.jl testsuite, maybe we can run the (small) polymake testsuite ( = examples) as well.
But that does not belong to this pull request.