Dev Meeting 2020.11.20 (opam repo testing and bulk builds)

Present: @kit-ty-kate, @talex5, @avsm, @thomasblanc, @rjbou, @MagnusS, @mseri, @samoht, @clarus, @emillon, @RalfJung, @pirbo, @mato, @patricoferris (notes by @dra27, edits by @avsm)

@avsm - purpose is for the opam team to be gather more information on uses of opam-repository. Recent meeting with @clarus on Coq's usage of opam-repository, this meeting follows up with Tezos (@pirbo) and Mirage (@samoht) and we'll demonstrate the new CI system for opam-repository (which has just gone live!). Gathering information on missing features which should be prioritised over the next 12 months.

@talex5 demonstrated the new infrastructure - there's an overview repo at https://github.com/ocurrent/overview that explains how all the components fit together.

Next steps:

• Extending obuilder to work on other platforms.
  • @patricoferris is working on macOS support
  • @talex5 working on FreeBSD support
  • @avsm working on OpenBSD support and ppc64be (big endian)
  • @dra27 working on Windows support

@avsm highlighted that all of this can be deployed on custom hardware - indeed, you can even have the deployment services on your own hardware and simply be submitting jobs to our cluster or have your own cluster infrastructure.

@avsm - slight audio lag problem; so let's go round-robin!

@clarus - big 👍 for the overview documentation! @avsm - could do with knowing what the docs story with Coq is at the moment - we're looking at docs.ocaml.org being deployed with the newer odoc later this year. Can contact Theo Zimmerman for more information on the Coq documentation side.

@avsm - switching over to Tezos, @pirbo. There've been some spectacularly big PRs - what can be being done to make those PRs less painful to get in. @pirbo - what's demonstrated looks amazing; health-check being done at Tezos is relatively naive, and it's clear that these tools might be better. Pain point at the moment is that the next release of Tezos requires Rust, and it's not clear how to integrate with the OCaml release tools. Being able to integrate these tools with GitLab instance would be great - @avsm there's no GitLab bridge at the moment, but that should be a good beginner project to contribute. Updating docker-base-images with Rust support should likewise be relatively straightforward. Using depext and automatic generation, it's possible to wrap Cargo packages in an opam repository. @kit-ty-kate - cf. https://github.com/ocaml/opam-repository/pull/16429 @rjbou - cf. also https://discuss.com.org/t/5743

@mseri - increasing issues for libraries depending on R as well, for example https://github.com/ocaml/opam-repository/pulls/16933

@mato - what's the status for custom pipelines, for example being able to put a generic pipeline a la Travis, AppVeyor, etc. @avsm - at the moment trying to solve specific problems (in particular, to get fast response times from CI). @dra27 - with tools and libraries today it's possible to write any pipeline using ocurrent - what we haven't written at this stage is the ability to drop a custom .yml file, for example, in your repo and convert that to a pipeline.

@mseri - can this be easily deployed internally, for example, to experiment at Citrix, or does it require a large cluster? @avsm (with @dra27/@talex5 nodding) - no, the whole thing can be done on one machine perfectly well; the cluster is simply for parallelising.

@avsm - ultimate aim with a lot of this - from opam-repository's perspective - is to increase the quality of PRs coming in because it's been possible to run the CI beforehand. Indeed, the hope is to be able to link opam-repo-ci into the individual ocaml-ci - so CI runs can confirm if the repo itself is still releasable to opam-repository (which, for example, will allow CI to confirm that pins are still working).

@avsm - @thomasblanc, we've talked before about porting Camelus over? @thomasblanc - at the moment Camelus is unfunded so it's just in maintenance only. It feels as though it's the grandfather of CI waiting to become obsolete - @avsm less so, it's useful to have the bot for interaction.

@samoht - how complicated is it to add a new service for deploying? @avsm - it's easy as long as the deployment can be deployed as a docker-service. Other plugins can be written for ocurrent-deployer. We've presently got obuilder as a simplified and domain-specific version of docker-build - it does feel as though we could do with orunner as a dual for deploying. @talex5 clarified that the deployment pipeline currently only keeps the services deployed, it doesn't handle the initial service configuration.

@avsm - coming up on an hour, we've not mentioned Windows! opam 2.2 during next year is the end-to-end Windows release and that should be being integrated into this. @dra27 - note that the support for Windows workers in the base images and ocluster is not gated on opam 2.2 and should be available early next year.

@MagnusS - preliminary system for ocaml continuous benchmarks in progress. https://github.com/ocurrent/current-bench

@avsm - wrapping up. Various follow-ups for this - aim to have a full follow-up meeting perhaps in around 6 months.