2020 Developer Meetings

11 Dec 2020

A brief dev meeting checking the status of 2.1~beta4

Beta4

@dra27 - Discuss post for CLI versioning (check for a previous one)
@AltGr - PRs/issues to unblock for beta4: 4429 & 4373
@dra27 - review 4427; assist unblocking 4405; check over and finish off 4451; review 4435 (for English); review and merge opam-file-format/pull/35

Aim for opam-file-format 2.1.2 release Monday/Tuesday and opam 2.1~beta4 on Thursday. Most of us then off for Christmas!

27 Nov 2020

Agenda

opam 2.1 beta4
Other things
- hidden-version issue with OCaml 4.12 and actions for 2.1

Notes

opam 2.1 beta4

opam-file-format 2.1.2 with the various fixes.

hidden-version

Rename hidden-version to avoid-version (opam-repository to use both just for the existing packages)
Post-process solutions to see if packages have changed to be hidden and warn (with a default of not installing)
ocaml.4.12.0 needs to be tagged hidden-versionh

https://github.com/ocaml/opam/issues/4452

@AltGr maybe earlier in opam 2.0 it might have been better to have an explicit flag to build without a solver might have been better, but seems too risky to change it now.

@Kate - related, but the "From Sources" info on https://opam.ocaml.org/doc/Install.html includes links to opam 1.2.2 and 2.0.2! @dra27 - oops, let's update that just to refer to the "full" tarball and link to the releases on GitHub.

Travis

We need to move the jobs from Travis to GitHub Actions during December (@rjbou may have a look next week).

13 Nov 2020

2.1 beta4

Painful issues for 4.12:

#4354 - @AltGr fairly sure already patched.
#4373 - critical for @kit-ty-kate @AltGr working on it.
opam-file-format 2.1.1 release ready
@rjbou wants to keep 2.1.1 as strictly the endpos - small collection of features for 2.1.2
#4427 - @dra27 check the new code and bump to beta5 if necessary

2.0.8 backports

Work on #4394
No plan yet for releasing 2.0.8, it's just collecting the work to release

New ocaml-variants layout

ocaml/opam-repository#17541
Experiment (which can be pulled)
It's certainly not the ultimate solution, but the UI presented is no worse than the present one (ocaml-variants.4.12.0+options,ocaml-option-flambda instead of ocaml-variants.4.12.0+flambda, for example)
If accepted, it may well be that we don't back-port it to older compilers, although @AltGr noted that having "virtual" packages for the old variant switches shouldn't affect the reduced pressure on the solver.
@AltGr anticipating that the effect on the solver is considerable, as the disjunction of available compilers is much smaller, but not actually got figures yet
@dra27 noted that +trunk will hopefully fold into dev-repo after this (@AltGr noted the UI is still not brilliant for this, as you can't create a switch at pin creation)

opam file downgrade plugin

Not clear when this is going to be worked on in.

30 Oct 2020

Agenda

Specs
- @dra27 opened https://github.com/ocaml/opam/issues/4394 to track lexer change. Is this worth implementing before opam-file-format 2.1.1 is released?
- @AltGr - has there been a chance to look into a prototype plugin for downgrading opam files?
opam 2.1 beta3: release status
Other things
- Testsuite improvements
- Package explosion experiment
Old issue sweep: see if we can get below 270 today!

opam-dev meeting 30-Oct-2020

@dra27 - should we include the changes to enforce opam-version: opam-file-format 2.1.1 to include change. @rjbou - would prefer a single release with just the endpos change.

@AltGr not had a chance to look into the prototype for downgrading opam files - will see in a fortnight.

opam 2.1~beta3

@rjbou - CLI versioning PR ready for revised version of CLI versioning PR. Everything else still blocked on that.

@dra27 - https://github.com/ocaml/opam/issues/4408 suggests that beta2 is reporting as beta3. If that's definitely correct, let's bump to beta4 to avoid confusion. @emillon - is the release procedure documented? @AltGr - scripts are in release/ and doc in doc/release @emillon/@samoht - looks like some of the post-release steps want updating.

@rjbou - beta2 was mis-tagged (it's tagged the version bumping commit, not the one before). @dra27 - so next one will be beta4.

Other things

Testsuite

@AltGr - Makefile tests have been completed upgraded to Dune cram tests. We don't get the nice coloured output, but having the changes with the code changes is of course much superior.

@dra27 - might be an idea to get opam-rt working as a submodule and the tests activated through Dune. Not urgent, but will almost certainly improve Windows story too (since opam-rt doesn't work on Windows).

Package explosion experiment

@dra27 struggled both to take notes and take part, so this is sketchy instead with a summary!

@AltGr - took @kit-ty-kate's experiment from July a bit further. Downside is security - at the moment ocaml-base-compiler is guaranteed to be a variant-free version. There's a problem with this, for example, if something pulls in a new package and the UI is not as clear for when that pulls in.

@emillon - is this more a UI issue, for example a warning that the compiler is changing.

...

Problem is being able to lock the switch so that nothing can change it. @AltGr - idea of only-* packages.

@emillon - how about keeping ocaml-variants and putting all this in a new compiler package completely. So you opt in completely.

Conclusions:

ocaml-base-compiler should remain a "vanilla" compiler
Introducing a new package (e.g. ocaml-configurable-compiler) breaks the ability in opam 2.0 to say opam switch create 4.11.1 (this is already brittle w.r.t. system compilers, but this would make it systematically impossible).
Slightly nervous that the increased power of being able effectively to depend on compiler options might have unexpected consequences.

@dra27 - possibly do this for 4.12 only? @dra27 to liaise with @Octachron about this plan.

So the plan:

Have ocaml-options-* packages for each of the available options (flambda, 32bit, afl, etc.)
Generate ocaml-options-only-* packages for each combination. ocaml-options-only-foo-bar depends on ocaml-options-foo and ocaml-options-bar and conflicts all the other ocaml-options-* packages. i.e. we have the combinatorial explosion, but it doesn't need updating with each version and the packages are definitely mechanically generated.
Do this scheme for the 4.12.0 alpha/beta/rc releases (with an announcement) if upstream is happy and then if the experiment looks to have worked, backport to old ocaml-variants packages after 4.12 is released.

Community meetings

@rjbou noted that only next week's has a date! guix meeting is up in the air as to whether necessary. Propose that having one of these every other week is working quite well.

Propose opam repo testing meeting on 20/11/2020 and esy/drom on 4/12/2020.

@dra27 to liaise with @avsm over the odoc meeting.

16 Oct 2020

Agenda

New(ish) specs
- opam file versioning spec (Spec-for-opam-file-versioning)
opam 2.1 beta3: status
- PR blockers:
  - #4385 - blocked on @dra27 review
  - #4071 - --download-only flag (@Armael) - for beta3?
  - #4376 - using dune to run reftests (definitely in beta3 milestone) - for beta3?
  - #4387 - placing opam file for the package in the build directory (@lefessan) - for beta3?
  - #4388 - post-install-commands modifying installed files - ready to go?
  - #4391, #4302, #4298 - @rjbou PRs
  - #4379, #4369 - @AltGr PRs
- Issues marked for milestone: see https://github.com/ocaml/opam/issues?q=is%3Aissue+is%3Aopen+milestone%3A2.1.0%7Ebeta3
Old issue sweep
- List from @dra27 for possible closure:
- Some time today going through some of the older issues!

Minutes

Present: @samoht, @dra27, @rjbou, @AltGr, @emillon

opam file versioning plugin

@dra27 introduced the spec.

@emillon - good to avoid the mass updates (pain with opam 2.0). Potential problem with versioning bug fixes.

@AltGr - for 1.2.2 -> 2.0 upgrade we did this with repository rewriting (which meant you couldn't use the Git remotes). The plug-in approach mitigates this.

@dra27 - can the redirection be based on revision number? @AltGr - think so.

@dra27 - we'd previously mentioned including the repo file in Git remotes to allow for caching. @AltGr - actually the redirect field is the bit specifically ignored. Could consider the possibility of the default branch being 2.0-compatible and using a > constraint to redirect newer clients.

@samoht - 2.1 could have a better field to display a better version error. At present, the "message" is a hack in the available where you put a-package-which-says-you-need-to-upgrade! @AltGr - it's late in the process in the to do something which would alter the format. @AltGr - perhaps best rewrite is available: opam-version > 2.1, etc. since then the message is quite clear.

@AltGr - plugin needs to be something opam knows about. If using the Coq repo, for example, you wouldn't have access to the main plugin.

@samoht - is there a halfway where this could be done without alteration to opam at this stage? e.g. with a normal plugin you have to use manually? @dra27 - not sure that you can hook it in the correct place at the moment, e.g. for opam-update. @rjbou - some work at the moment for wrapping around opam.

@AltGr - @lefessan having similar issue for opam-bin wanting to rewrite opam files before they're installed. @dra27 - could we be looking at an additional hook?

@dra27 - this will also need some care with Connex. @AltGr - it should be fine because Connex is called to verify the repository before all other processing takes place. @AltGr - a simple approach could be to have a special plugin which is called automatically by opam to be called whenever an opam-version it doesn't understand is encountered. @AltGr - could experiment with this. @dra27 - need to consider whether updating the lexer to enforce opam-version being the first line. This is worth having in 2.1?

Actions:

Lexer/parser updated for 2.1 to ensure that opam-version is the first line for opam-version > 2.0 (@dra27 will open issue)
@AltGr look into the PR for a simple plugin called on-the-fly and then a decision can be taken on whether it's 2.1 or 2.2

opam 2.1 beta 3 status

#4036/#4071 and #4355/#4379 are both blocked on #4385 but are ready to go.
#4376 - nice to have. @emillon - views on further tests using Dune? @dra27 - existing tests are only written in make since it's older than Jbuilder/Dune. If ready, then it can go in (completely safe, as it's testsuite) otherwise no problem to bump to next milestone.
#4387 + #4388 - #4388 is ready to go; #4387 is ready to go, although it would be better to have a variable for where the opam file is written - @rjbou happy to look at this next week
#4298 + #4302 are both queued and ready pending opam-file-format (which needs https://github.com/ocaml/opam-file-format/pull/24) and opam-file-format release
#4332/#4250/#4369 - @AltGr reasonably confident that it's ready in that it shouldn't introduce new bugs. There are some hidden environment variables which can control the solver which aren't documented, but this should be OK - they allow a certain level of debugging to be possible if there are issues. This should be fine for beta3
#4315 - @rjbou on this one; partial fix expected for beta3
#4323 - slight (necessary) bikeshedding on exact name, but fundamentally blocked on the CLI versioning infrastructure
#4321 - similarly blocked
#4320 - 2.1 features look fine; can be removed from milestone (the issue as a template might want to move to the wiki for future plugins moving in opam)

Possible readiness for beta3 end of October / early November. Review the items in the 2.1 project which aren't in the beta3 milestone and decide which ones to address, but aim would be for a relatively swift transition to releaser candidate.

Old issue sweep

Handful of old issues gone thorough in the last 20 mins or so. @dra27 to action a few further closes with a message.
@dra27 will create a project to house "Contribution of this welcome, but it won't ever be implemented otherwise" issues.
This is possibly a better way of doing this than having a dedicated meeting to issues - using the last 20 mins or so to sweep a few at a time, and we'll keep chipping away!

2 Oct 2020

Present: @altgr @samoht @rjbou @avsm @dra27 @emillon

Goal of first security meeting on Oct 9th is:

hooks as a role
timelines for tools
anything needed in 2.2
CI needs?

Notes:

etienne/samoht many different models of security
it is that every client gets the same binary and archive
emillon: work on conex has been done but not deployed due to ci
avsm: needs CI infrastructure via ocurrent
altgr: conex adds as little friction as possible by design but we need the tooling like in opam-publish
dra: check is to make sure package doesnt break the repository
we should invite kate/mseri/talex5 after this first meeting to discuss the impact on the opam repo ecosystem and CI
output is integration roadmap, and impact on opam ecosystem
avsm: can we push opam admin add-hashes?
altgr: hashes are all checked and first one is key
avsm might slow down the full repo calculation on 32 bit due to sha256, need to sanity test a rPi3
bug hunting - need a morning reserved for a bug sweep. (2nd October).

New calendar:

opam-bin/conex/orb - @rjbou to invite for Oct 9th.
- security and reproducible builds
- uses opam hooks, so we could unblock development in opam 2.2 without disrupting client roadmap
- Confirmed attendees: Fabrice Lefessant (opam-bin) Hannes Menhert (conex/orb)
opam2nix/debian/rpm (proposed Oct 23rd)
- @rjbou knows some guix maintainers as well, who might be interested.
- Confirmed attendees: Tim Cuthbertson (nix/opam maintainer) , Patrick Ferris
- Invite jonludlam , stephane glondeau, mehdi d
- timezone problem: GMT+10 / GMT-4 / GMT+1
- future meeting with Julien Lepiller (guix/opam maintainer) as in NYC timezone
persons who manage the coq-opam-repository, (proposed Nov 6th)
- @rjbou knows the right people
- invite opam-repo maintainers too
- to invite: emilio
- Confirmed attendees: Guillaume Claret (Coq opam repo maintainer) Ralf Jung (iris CI maintainer) Kate Deplaix (opam repo maintainer)
opam repo testing and handling and bulk builds - plan December
- review current testing strategies and glue them together
- armael - marracheck
- kate/mseri - maintainers
- talex5 - ocurrent, 0install-solver
- magnus - ocurrent
- @rjbou: Guillaume Claret & Ralf Jung could be interested, both CI people
docs.ocaml.org
- @jonludlam is making progress on the model rewrite
- Thomas Blanc also has bits of glue and is sharing this work with Jon
- will find a slot in an opam-dev meeting for this rather than opam-dev, with us attending.
esy/drom (@Altgr to invite, Date TBD)
- alternative opam clients
- invite @andreypopp and Fabrice
- also invite talex5 to discuss 0install.
We need to pull this calendar out onto the opam wiki separately from these meeting notes

25 Sep 2020

Present @dra27 @emillon @avsm @altgr @rjbou @samoht

A morning of calendaring. Since we're over the opam 2.1 beta hump, this is a good time to plan out the community meetings we need to ensure we hear from all the contributors. Here are some meetings and proposed dates (subject of course to the various invitees being interested and available).

opam-bin/conex/orb - @rjbou to invite for Oct 9th.
- security and reproducible builds
- uses opam hooks, so we could unblock development in opam 2.2 without disrupting client roadmap
- Confirmed attendees: Fabrice Lefessant (opam-bin) Hannes Menhert (conex/orb)
opam repo testing and handling and bulk builds
- review current testing strategies and glue them together
- armael - marracheck
- kate/mseri - maintainers
- talex5 - ocurrent, 0install-solver
- magnus - ocurrent
- @rjbou: Guillaume Claret & Ralf Jung could be interested, both CI people
opam2nix @avsm (proposed Oct 23rd)
- @rjbou knows some guix maintainers as well, who might be interested.
- Confirmed attendees: Tim Cuthbertson (nix/opam maintainer) Julien Lepiller (guix/opam maintainer)
- timezone problem: GMT+10 / GMT-4 / GMT+1
persons who manage the coq-opam-repository, (proposed Nov 6th)
- @rjbou knows the right people
- invite opam-repo maintainers too
- Confirmed attendees: Guillaume Claret (Coq opam repo maintainer) Ralf Jung (iris CI maintainer) Kate Deplaix (opam repo maintainer)
docs.ocaml.org
- @jonludlam is making progress on the model rewrite
- Thomas Blanc also has bits of glue and is sharing this work with Jon
- will find a slot in an opam-dev meeting for this rather than opam-dev, with us attending.
bug hunting - need a morning reserved for a bug sweep. (2nd October).
esy/drom (@Altgr to invite, Date TBD)
- alternative opam clients
- invite @andreypopp and Fabrice
- also invite talex5 to discuss 0install.

other discussion

opam2web
- everything is very old
- not using camlp4 but needs a refresh
- @avsm to have a go at refreshing it for ocaml.org
add a repo archive-mirror by default?
- when using a git repo the repo file is ignored because we specifically dont want to use redirects
- can we configure a global archive mirror for opam 2.1 by default
- @dra27 would be immediately useful if the repo file were read in 2.1 so travis will work
- @altgr action an issue
md5 deprecation
- @altgr to do an opam admin to rewrite opam repo
- @avsm to apply it to opam repo

18 Sep 2020

Present: @dra27 @altgr @altgr @rjbou @etienne @avsm @Nathanreb

welcome to @NathanReb
Agenda
- opam-monorepo
- opam-compiler
- opam-tools
opam 2.1beta3
- post dependencies and cycle with dev dependencies
- opam-clean
- memory tracking
- cli versioning actually doing 2.0 compat
opam 2.1 point release
- plugin for version tracking
opam 2.2
- hooks for opam-bin?

opam-monorepo

@nathanreb: turning duniverse into an opam monorepo plugin. It generates an opam lockfile where all the deps are locked to a specific git ref. it allows opam to efffectively skip the solving part and pick concrete versions. The plugin also allows all the sources to be fetched locally and built with a single dune build invocation. It allows the full set of opam commands (remotes, pins etc) to be used, while supporting a monorepo plugin.

@samoht: very close to opam lock, but it doesnt use the packages from the local switch. instead it looks at the existing opam-repo (and switches) instead of the local packages. So this is a complementary solution to the opam-lock.

@altgr: main difference is that opam-lock supposes that we have a working build environment and want to share it with someone else. the opam-monorepo creates a dev environment from the spec from scratch and offers the possibility to create a dune hierarchy. How do you decide what to include in the duniverse? (e.g. do not include ocaml itself).

@NathanReb: have a list of packages that are considered "virtual packages" (eg. ocaml or dune or base-*). Besides that everything gets pulled into the dependency tree. The lock step only accepts packages that build with dune at the moment. Used to allow this via duniverse opam-install but it is complex in the presence of cycles.

@Altgr: go down the route of analysing ...

@nathanreb: we could also snip out the dune requirement and pull the filtering logic more generally. @AltGr: this should generate a re-usable opam file. @avsm: can we fix the pin issue for 2.1? @NathanReb: just a performance issue. @rjbou: revisit next week.

@samoht think about the naming; we could fold in the lock part of monorepo to opam-lock. and opam source could be extended to work in parallel and remove the need for the separate plugin. @emillon the name monorepo has little to do with its function. @altgr this is more related to vendoring, so perhaps call it something related to that. @altgr opam source --deps makes sense @NathanReb its important to have the workflow in one place to solve the problem at hand and then generalise it. For now, its good to remain as a plugin.

opam-compiler

etienne has done a release!
some of nathan's concerns from the previous item are shared with opam-compiler.
plan: we will treat the workflow plugins as an experiment in opam 2.x, and keep an eye on what the community does with it.
@avsm to forward to caml-devel

opam-tools

not sure where the patches are for fdopen's fork to find out the windows rsync subpath bug. are local switches just broken in the cygwin fork?
dra27 could switch to robocopy instead which does differential copying
as a dev team, we need to switch to Windows for our primary platforms
@emillon: the goal is to provide fast switches and opam-tools is one way to achieve that. what are the other techniques to make this happen? compiler support not there yet, binary rewriting as esy does. Is there something we can do a little faster?
@altgr - ocp-bin also exists. uses hooks which were also intended for use by conex.
@avsm - let me ask on caml-devel about opinions about binary packages
dra27 todo opam discussion
@emillon even backporting the patch to ocaml-secondary-compiler could go a long way

opam 2.1 beta3

cli versioning actually doing 2.0 compat
- cli interface is merged but doesnt do anything
- dra27 has a todo list of things that have to be done - @rjbou to look at this. @dra27 to create an issue.
post dependencies and cycle with dev dependencies
- @emillon several slightly different problems.
- first one is related to depext. We don't support dependency cycles in opam, but we consider that its ok if one of the edges is tagged with-test, as it can be broken by installing main deps first and then test deps. However, the tools cannot parse this correctly and so depext fails. This is fixed in opam 2.1 but is a problem in 2.0. Is more visible recently as such a cycle has been added to dune with csexp and ppx_expect. Also a similar issue in irmin.
some people have suggested using post dependencies to get over this. this sort of works, but all tools have to support this -- for example, ocaml-ci.
these post dependencies havent hit opam-repository yet - if there is a way to solve this without them then that would be good.
option 1: recommend opam 2.1 everywhere. should there be a policy around post dependencies.

@altgr: not sure that post dependencies are a good fit in this case, as post-test deps might only get installed after your tests get run. So this really doesnt work for the test usecase. The real solution would be that opam separates the build and test steps. A bit weird to have the test step after the build step, but thats ok.

A workaround is to use a custom variable as a filter for the dependencies. By default these tests would never get tried and then manually set the flag when they need to run. Problematic packages could use a custom variable.

ocaml-ci could do a different scheme: solve for both with and without test and generate dockerfile. @altgr wont work as the revdeps will be removed when reinstall -t is run.

recommendation:

create a separate test package for now.
or add a flag for now as a custom flag
@rjbou: https://github.com/ocaml/opam/issues/4099

opam clean

posibly a bug that build directories arent removed
sources can be removed if packages not pinned
crunch

Future meeting plans

Need to plan out future opam dev meetings, and ensure we invite new projects regularly.

Initial invitees (followed by discuss post)

fabrice - opam-bin - @altgr to do so
coq - invite (emilio?). persons who manage the coq-opam-repository, who is in charge of the CI. @rjbou
opam2nix - @avsm
docs.ocaml.org - perhaps defer this to the odoc-dev meeting? @avsm to discuss with @jonludlam
bug hunting - need a morning reserved for a bug sweep
windows - after opam 2.1 rc is out

11 Sep 2020

Present: @AltGr, @emillon, @avsm

Notes

compiler plugin
- @NathanReb to help out, and the first version has been announced on discuss and sent to opam-repo!
- initial version which does switch creation, and more features to follow
- works with 2.0 and 2.1 opam as it only shells out and not call opam-lib
2.1 beta status
- 2 severe bugs in beta1. first check for sandboxing was broken so it breaks opam init. Scripts werent initialised in order and so scripts would disable sandboxing. Now fixed in beta2, so delaying more public announcement until then.
- @avsm is working on ocaml.org refresh and so will report on opam.ocaml.org merging soon
- Solver invariant failures were too verbose and that is now fixed in beta2 as well.
test deps and opam depext
- opam depext -yt installs the root package and test dependencies which is causing packages to fail
- this is currently causing failures in Irmin and other projects.
- looks like a quick fix in opam-depext is quite invasive, so this will have to be a 2.1-only fix for CI
- @avsm notes that the CI containers do include opam2.1 snapshot binaries as well
- would be useful to add opam 2.1 installation option to the github actions for ocaml
Platform plan:
- as per OCaml Workshop keynote, @avsm is arranging to put the slides online at ocaml.org
- taking a bit longer as infrastructure needs a refresh
- creating a library to represent incubation status of tools
- proposes to make the last opam meeting of every month a wider Platform meeting, with lead maintainers attending and an update of the broader picture. Minutes from this meeting can form the "Platform monthlies" on discuss.

4 Sep 2020

A very quiet meeting due to holidays. Just @emillon @avsm @samoht present.

We primarily discussed the way forward for the Platform, and the re-discussion took place the week after in https://github.com/ocaml/opam/wiki/Dev-Meeting-2020.09.11

28 Aug 2020

No meeting due to the OCaml Workshop 2020 talks.

21 Aug 2020

Present: @avsm @rjbou @altgr @samoht

@dra27's opam cli spec and PR are now up, and look good. @Altgr and @rjbou will review this week to get it into 2.1 beta
@avsm reports that the OCaml Workshop platform talk is recorded, and reflects the roadmap discussions on the earlier dev meetings for the 2.2 roadmap

semantic versioning

Looking forward, @avsm wonders how we can progress the semantic versioning discussion and figure out what to do about upper bounds.

@altgr we have a spec to add lower and upper bounds; it uses existing opam-file-format and only works at the lower solver levels.

@avsm would it work if we just bounded on the major version number?

@samoht don't think so, as he is not convinced by semver, due to the module type problem - any module exposed could be used by someone else.

@altgr: thats why having an operator which leaves the possibility of using it is a good middle ground. no real blocker, but we risk confusion if we diverge from debian's definition of semver.

pin depends slowness

@rjbou to look at pin-depends for 2.1:

perf is slow, can be parallelised
https://github.com/ocamllabs/duniverse/pull/110 shows lock file problems
there is an issue opened by @NathanReb on opam about this now.

syntax evolution

@avsm Post 2.1, we need to start to think about how to work towards a common syntax library with dune and ocamlformat, while maintaining back compat.

@altgr opam uses the basic ast parser and then a higher level parser, so this could work at the lower level.

@samoht when you write formula the version constraints are difficult to transform, and also we need a new quoting mechanism.

@avsm similar to dune which uses :var for quoting bare vars.

This seems like a difficult problem with unclear rewards at this time. Will probably defer till after the 2.2 "windows" release.

2.1 beta

No particular blockers; the project is uptodate thanks to @rjbou
@altgr has generated new project keys for the release, will share securely with rest of dev team.

14 Aug 2020

Present: @rjbou @emillon @avsm @dra27

Most of the meeting was spent going through the opam 2.1 alpha3 blockers, and discussing infrastructure. Alpha3 is due to be released this week. No substantial decisions other than those reflected in the issue tracker.

7 Aug 2020

Agenda:

review blockers:
- opam-file-format release PR: dra27?
- disable sandboxing PR: witing for dra & mato
- infrastructure move, can we disable logs (@avsm)
opam cli spec
ocaml compiler plugin
opam bin
opam 2.1: status
- @avsm can we not include opam-installer in images?
- 2.1 alpha3 scheduler
opam 2.2: recap features/roadmap for OCaml Workshop talk
progress on opam solver survey

Notes

Present: @dra27 @rjbou @emillon @avsm @altgr

Blockers

opam-file-format release blockers

PR #23 simply needs reviewing and merging. Deferring the CRLF one till opam 2.2+

sandboxing blocker

@dra27 - not been able to investigate the WSL/WSL2 detection satisfactorily. Suggest proceeding with the PR as-is - the detection can improve it later by suppressing the prompt.

infrastructure blocker

Logs are a potential GDPR problem - for the time being, best thing seems to remove the logs and remove the "popularity" tabs. We will replace them with objective (non-log-based) functions in the future, for example using dependency cones rather than download statistics.

opam CLI spec

opam plugins issues

@avsm Story for plugins supporting both opam 2.0 and 2.1 is still awkward since a binary linking to opam-libs would initialise state and then silently upgrade ~/.opam to opam 2.1 format (even if the main binary is 2.0). @dra27 - potentially can the CLI versioning help here - require the latest version of opam and use CLI versioning for users who need opam 2.0-like behaviour.

@avsm Could (should) consider using a different root certainly for betas - or possibly even a different root per version?

compiler plugin

@emillon - updating working on having an executable which can run a command, track and update the changes. This allows the original aim of being able to update the compiler in-place, but it's not clear how it'll integrate with the rest of the plugin.

opam 2.1 status

@avsm - move opam-installer to a separate package in a separate repo (i.e. like opam-file-format). Going to remove opam-installer from the container images as our own binary installer doesnt use it anyway.

@dra27 - it depends on opam-format, so it makes less sense to move it to a separate repo unless it's rewritten.

For now, simply not build opam-installer by default - so released packages are the opam binary only.

alpha -> beta status (the github project is all uptodate):

Sandboxing
opam-file-format release
switch invariants fixes
opam CLI versioning

Can release alpha3 this afternoon. No announcement needed, it is primarily for the opam-tools plugin to be able to use the 0install-solver.

Platform slides

opam 2.2 - largely based on @dra27 Windows presentation, as we agreed that the features generalise really well. Most of the talk will be about the opam 2.1 advances anyway, but we'll do a roadmap for the first time this year.

31 Jul 2020

Present: @rjbou @dra27 @altgr @avsm

opam-tools

@avsm - only thing blocking it at the moment is getting a fast solver working for empty switches. Need to get the opam files for the packages visible in the current switch - @rjbou possible from the API. Problem with using the API as the 2.1 API (vendored in opam-tools) will upgrade a 2.0 switch. @dra27 - vendor both APIs and select between them in Dune?

It would be helpful with the 0install solver is integrated for these - @dra27 enabling solvers are low-risk (since they're not enabled by default and you can always configure opam with diverse solvers, so you can never depend on a version of opam implying the availability of a specific solver).

2.1 beta

@rjbou wants the sandbox detection work merged (@dra27 agrees, and working on WSL detection to allow the prompt to be suppressed)

Also a CLI pinning inconsistency to fix, which @rjbou has a PR for.

@dra27 finally working on CLI versioning and will have a spec for review next week.

@avsm - can we branch 2.1? @dra27 - pain while we're still adding features. Branch at beta instead? @avsm that's fine, we will adapt the container builds to go from master. Just don't commit any non-opam-2.1 features at this stage.

@avsm Previous versions (including 2.0 - i.e. LTS) will go into security-only updates once a new version comes out, since the state of the root isn't forwards-compatible (i.e. 2.0 client can't do anything once you run with a 2.1 client).

24 Jul 2020

Present: @rjbou @dra27 @avsm @emillon @altgr

opam switch create

@avsm noted coming from opam-tools plugin the problems with the solver timing out - the issue is that the solver is having work before the ocaml package version has been locked. Re-exploring the question of whether the 0install solver should be integrated in 2.1. @AltGr has been looking into it. @avsm - we had been talking about having different default solvers for switch create and install, but this isn't necessary - the opam-tools plugin can simply specify the solver manually, if the support is there.

Cygwin local switches

@avsm struggling with @fdopen's opam with local switches. @emillon also having related issues where local switches were symlinked, but opam normalises them.

Failure mode is simply opam switch create . in Cygwin opam.

opam clean

The Coq team seem to be trying to come up with Docker containers where they can reduce the state of the opam switch to nearly 0 and then being able to redownload/reinitialise it later (when it needs updating).

Does the problem come down to source pinning - build directories remain at the end? Doesn't look like features needed for 2.1, but there could be changes for a 2.2+ feature.

compiler plugin

@emillon working on other things this week - some feedback from @lthls, though not yet from @ctk21. Knowing exactly what to do when changing the layout of installation is tricky, at present there's no way in opam just to run the installation command and update the .changes file. The other problem is that the plugin has started on the premise that it would operate on special switches for each type of use-case (based on a PR, based on a path, based on a PR, etc.). Problem has arisen is that the reinstallation can only be done for certain kinds of switch - possible solution to have a general command to replace the compiler with one just built.

alpha2 ready to go!

ocaml/opam#4240 added back into the alpha2 milestone, but potentially this could simply be in alpha3?

CI

@rjbou working on automatic disabling of sandbox for CI. Should have something for alpha3 where the sandbox is tested and either the user is prompted or opam detects whether the sandbox should have failed (e.g. WSL, unprivileged docker container)

17 Jul 2020

Present: @dra27 @rjbou @avsm @emillon

opam-tools update

@avsm reported that the plugin is working; it creates switches quite quickly. How do you publish it to npm? i.e. what happens if it's run in an environment where opam hasn't been run before?

opam-tools at present creates 3 switches: compiler switch (virtual system compiler); tools switch and then a local switch containing the deps. Question as to whether that should be automatic? @dra27 reckoned it should be at least controllable; @rjbou suggested that the tool should be able to show the commands it would run (then allow customising).

opam-compiler update

@emillon on vacation for most of the week, so limited update. Completed documentation and sent to @ctk21 and @lthls for comments. @ctk21 certainly happy with features.

@avsm - can plugins depend on other plugins? In particular, can opam-tools therefore use opam-compiler?

@emillon Also looking into feature request from @lthls for transforming the compiler - for example, switching from a compiler built with world.opt @dra27 suggested that this may want pushing back into the compiler (for example, having selective uninstall targets) rather than the plugin going overboard.

alpha2 update

@rjbou - most PRs merged; two remain for reviewing.

Milestone not in sync with project (bloomin' GitHub!), so we have to curate this manually.

Question from @AltGr with ocaml/opam#4252 - rather than removing the recursive & subpath pinning, could possibly be left in as experimental?

@dra27 prefers ocaml/opam#4265 to be in an alpha-quality release. Question over the change in CI - @rjbou expects it should be reasonably easy to use @AltGr's recent work for setting up mirrors of opam.ocaml.org to have a version where you can request a specific SHA by https so that the CI can also use the http backend.

@dra27 will open an issue to track it (concerned that the http backend won't be getting any testing, given that it's the default).

explore.ocaml.org update

@patricoferris's work with @avsm on scraping the Notion output to allow conversion to a basic site.

@avsm considering documenting the architecture of the platform as well - i.e. how VSCode actually works for OCaml (i.e. how it pulls in lsp-server, how that then uses Merlin + Dune, etc.).

@emillon - concern that it would grow stale. @avsm trying to organise with @patricoferris monthly review of sections of it.

@emillon - should promote the USP for this - i.e. why does this resource get updated vs Awesome OCaml; ocamlverse, etc.

@avsm - aim for this is that the authors of the tools should be the primary authors of the workflows.

beta blockers

.opam/config needs to report 2.1 (issue needed)
CLI versioning - @dra27 to write a spec

10 Jul 2020

Agenda

Review last week's blockers
- (none)
Opam workflow documentation
Spec Review and Opam 2.2
- Lock file spec
- Depext spec
- Compiler spec
Discussion
- Release scheduling

Present: @avsm @rjbou @altgr @patricoferris @dra27 @samoht @emillon

Explore OCaml

@patricoferris presented initial work on a site to gather opam workflows. It is a more dynamic approach than ocaml.org and recognises that tool workflows evolve - @avsm proposes this eventually becomes explore.ocaml.org, but wants to show the idea to opam-dev first before going out more widely.

notion.so provides a mechanism for defining workflows allowing articles to be quickly cross-referenced. @samoht wondering if there's an API for it? @avsm - on roadmap, but they do have a CSV+Markdown export function which could be parsed in OCaml. @avsm hoping this can be used to aggregate the various blog posts to a central site, rather than workflow documentation being a distributed thing.

@dra27 liking the easier access to updating the content quickly (for new users)

@avsm/@samoht concerned about risk of lock-in, but may well be halfway house where content from ocaml.org can be moved over. @AltGr says "awesome", @rjbou gives approval. No negative feedback from opam-dev.

@avsm - next step will be to move this over to platform-dev (with Yann Regis-Gianas/Théo Zimmerman/Antonin Décimo) and then wider discussion. Needs HTML output first.

opam 2.2

Four feature areas:

Explicit dependencies
Package parameters
Layered switches
Build environments

@avsm reported on a prototype opam-tools plugin. Works by maintaining a set of "system" compilers inside ~/.opam with a minor tweak to ocaml-system to maintain PATH. It the creates a switch per compiler version which is used to build and cache binaries for tools and the user simply has to run opam tools and the binaries are copied - untracked - into the switch.

@AltGr experimented in the last week with hidden packages as an implementation of the "explicit" dependencies - the change is completely transparent to opam 2.0.

compiler plugin

@emillon - plugin development going well. Working on some requests from Vincent Laviron, in particular for reinstalling the compiler (so allowing configure options to be altered and the reinstall specific parts of the compiler).

Question on how to update the switch when - say - the upstream PR branch is deleted. @AltGr at present, when the package changes it is simply marked that way, but the old SHA is not kept - this could be usefully extended for the plugin. Should be possible to avoid the problem by pinning to the GitHub magic "pull" refs.

alpha2 blockers

@AltGr reviewing going well - should be ready by next week or the week after.

OPAM_CLI_VERSION proposal

@dra27 proposed adopting Dune's lang dune style versioning for the opam CLI. The idea is that you can either specify OPAM_CLI_VERSION=2.1 or --cli-version=2.1. @dra27 proposed adding the support for the option but with no difference for 2.0 and 2.1 - @avsm suggests going further with it.

Benefits would be stability for CI and so forth and increased ability for faster releases, since it should be easier to force upgrading.

3 Jul 2020

Agenda

Review last week's blockers
- (none)
Windows support review
Spec Review
- Lock file spec
- Depext spec
- Compiler spec
Discussion
- Release scheduling

Notes

Present: @rjbou @avsm @dra27 @altgr @emillon.

Windows support:

@AltGr: mini hackathon last year really helped. We should do that again. @avsm @dra27 in agreement, but just no idea when due to the COVID-19 situation! Would a virtual one work?
@dra27 "opam on Windows - getting over the line" presentation starts.
Had some ideas over where we were vs last year. The problem that Windows exposes is a huge number of choices. There are 4 officially supported OCaml distributions (MSVC, mingw-w64, Cygwin, LXSS/WSL), and the PE compilers exist in both i686 and x86_64 variants.
Right now, opam's build environment is the PATH, and on Windows there is the option of WinSDK, Cygwin, MSYS2 and WSL. It is unusual for software to install anything into Windows' system paths, and there is no universally blessed package manager. @avsm does the new Windows package manager change things? @dra27: it adds more choices, and is a "competitor" to choccy. The Windows manager looks more geared towards application distribution, and there appear to be no dependencies between packages. It appears to be a repository of "setup instructions" for binary packages, which is quite different from opam's source based approach.
Default shell on Windows also varies: cmd, Powershell, Cygwin, MSYS2, WSL. MSYS2 patches tend to be C header style patches and there are a lot of them. MSYS2 is very reactive to patches in terms of including them. WSL is largely a cross compiler.
These choices already exist in Unix, but they are internalised and ignored! @avsm: and if we do a dedicated Windows release, then it will benefit the sprawl of Linux/macOS compiler options as well.
Windows needs:
- Switch invariants (in 2.1)
- depext lifting to availability (in 2.1)
- Explicit dependencies
- Package parameters
- Layered switches
- Build environments - encapsulation of where you are going to build these packages.
@avsm flexdll: can we bundle it into OCaml itself? @dra27: it is used by other projects and fixes a specific shortcoming of Windows (unable to look for previously loaded symbols in shared libraries).
explicit dependencies: explicitly specify one that has to be added by the user (e.g. do not select flambda by default ever, unless asked with a switch create). @emillon: this looks like the "do not select beta versions" feature as well. @altgr: there might be problems with solving where you need intermediate steps to reach them. @dra27: dont think this will be the case; it just needs to be specified explicitly rather than searched for. @Altgr: need to carefully consider the impact of state reloading on performance here.
@dra27 we would gate this at opam-repository level. @avsm: very useful to have a flag for these elevated packages that
@avsm: can we remove depopts and make that {optional} as a dependency annotation. @dra27: yes can make this dual to {explicit}. @altgr: if you specify a depopt with a version constraint, then it makes the results of the solver confusing for the user. @emillon: activate optional things depending on a number of depopts, and this is a real problem. @avsm: we used that in mirage, and its a user nightmare -- been migrating away to a set of opam packages with a single depopt instead.
@dra27 on windows, we can use explicit dependencies to select exactly which compiler is needed. @avsm this is also useful for (e.g. beta compiler selection) and explicit toolchain selection on macOS xcode-select. Useful beyond just Windows.
package parameters: pass through options to opam packages and persist them around reinstall. we delete hundreds of ocaml-variants packages when this is done. @altgr .config files are currently transient, so do we need a new one? @dra27 it is possible to feed through values fairly easily, so the implementation is easier than expected. this also helps metacaml become a "real" opam package. @altgr: we need to be careful if installing a package which is in conflict with the system, and the parameters error message need to be much clearer.
@avsm why cant we just have a base-flambda package to detect and conflict against flambda? we could make conf-flambda depend on ocaml with the post flag, but it might make ocaml recompile. consensus: we could just add a base-flambda.
depext lifting: has been done in 2.1 and is important for windows support. This one is quick and easy.
Layered switches: not a critical feature for Windows, but very useful for tools bootstrap.
@avsm When we install an opam plugin, it is installed to a shared bin area. Could we use this to build isolated switch support? @emillon @dra27 @avsm to investigate as part of opam 2.2
@dra27 build environments: run everything in the system build environment with a sandbox. Generalise this feature into running a different Cygwin envioronment and indeed even a Docker container. Need to separate artefact extraction from build environments. @altgr could super sandbox to find missing implicit dependencies. @dra27 could have different backends a bit like the current repo backends, but need to be careful not to turn them into the lowest common denominator and make the common case (localhost) slower.
@emillon can we build the binaries reproducibly on windows? @dra27 should be as long as we build the same C compiler. @dra27 this is the relocatable compiler, and not a fully reproducible one (useful for setting up switches faster).

Consensus:

@avsm nothing in these features is windows specific. This is awesome! it will greatly improve the quality of life of opam users across all
@altgr we thank @dra27 for making Windows support interesting and exciting! many of these concepts exist already but are global and specialised to unix (eg bubblewrap) and this generalises this.
@rjbou also thank you @dra27 and looking forward to working on these.
@emillon also thank you for the work, and surprised a bit how little it is about windows.

26 Jun 2020

Agenda

Review last week's blockers
- depext/cygwin
Spec Review
- Lock file spec
- Depext spec
- Compiler spec
- Recursive pinning spec
Discussion
- Release scheduling
- pr backlog

Notes

Present: @rjbou @avsm @samoht @emillon @dra27.

Depext/cygwin:

there is a forked opam (on bitbucket?), and a forked opam-repository that exists for Mingw cross compilation.
also a forked opam-depext plugin which supports using mingw packages from cygwin, which is distinct from the cygwin support
there is a script which adds a package manager to cygwin, as that doesnt natively provide one (setup.exe is different and for one-off installation sets)
We could add the cygwin metadata to opam-repository without breaking anything, as that is pure depext rules.
What do we do if we do not detect the right OS? could we add a test case to make sure packages remain available. @rjbou to investigate.
Do not solve this for 2.1, but decide when to do an opam release dedicated to nothing but windows support.
dra27: will investigate and write up a feature spec over next few weeks about this.

Lock files:

retrieving the vcs hash via duniverse and create opam lock files looks possible. This has the advantage of not requiring a new metadata format for duniverse -- just reuse the perfectly fine opam format, which is possible thanks to lock file support. @avsm continues to investigate this.
when pinning have the hash as the version, could possibly use this functionality in the future. @rjbou has this code in a branch.
lots of different workflows for local development. need to figure out which ones are used and which ones can be deprecated. its not complicated, but we have too many choices. Its not clear for a new user what all these concepts are for. having a recommended tutorial with some initial workflows would help.
- Idea: create an "opam tutorial" plugin that does an interactive walkthrough for new users -- this could subsume user-setup. avsm needs to look at this for real world ocaml, so will start this spec off.
opam-0install-solver PR: looks self contained but pushed out to 2.2 to get more testing. Need to check that this causes no blockers. 2.1 is feature locked so nothing new until the existing release branch is cut.
emillon: opam compiler plugin: can now reinstall a compiler inplace by running the build and install commands. As a user need to think about where your switch lives and what the compiler is tracking. As a next step, we need something to detect if we have a compiler tree where you might want a local switch and collect where they want the switches to live. Ready to get some testing: will ask Tom Kelly from OCaml multicore team and Vincent Laviron from OCaml dev team.
- Where to store user data such as switch location? Can edit it during pinning ocaml-variants. Can change the OPAMEDITOR variable to do this non interactively.
performance improvements PR. This gives a huge improvement on opam switch list-available, especially with a lot of local switches. People who werent using opam1 and started opam2 use local switches a lot, and so they end up with a lot. This results in multi-second commands.
need a PR to remove the --rec flag to disable this in 2.1. rjbou: is now on the project todo for 2.1 on github projects.
dra27: ocaml version number changing to use debian-style. Where do we want +flambda and where should beta2 be. Need to figure this out in time for ocaml 4.12. avsm: also working on ocaml-variants to fix gcc10. Many many patches needed for compiler support at this time.

Overall: releases for opam post 2.1 look like they will be "big feature" tickets for each release. Future dev meetings will discuss specific targets (e.g. Windows support, and monorepo support, and improved onboarding). This will let us release faster and iterate better with more focus.

19 Jun 2020

Agenda

Review last week's blockers
- (none)
Spec Review
- Lock file spec
- Depext spec
- Compiler spec
- Recursive pinning spec
Discussion
- binary packages
- pr backlog

Notes

Present: @rjbou @avsm @altgr @samoht @emillon.

Lock Files:

isolated switches would solve things. andreypopp brought this up.
louis: how are isolated switches different?
avsm: they want to lock the opam-repo metadata as well, since that is a source of non determinism.

We can do this at the moment by:

create a switch with a specific URL to a repo
then use a lockfile
could add x-opam-repo
this can be done via a plugin
need to use the git repo
rename platform-blog to opam-blog
- rjbou information is scattered, e.g. graphics tutorial on ocaml.org is out of date and gives incorrect opam information since graphics split.
- avsm to rename repo and move to ocaml-opam org.
splitting opam lib to be easier to consume
louis: we could simplify this to some extend but not to interpret formula
rjbou to review the opam2nix PR as it looks in the right direction.

depext integration:

prs open now for 2.1alpha2
solver changes required so this needs review.
need review from louis, he will do this on 29th June onwards, so alpha2 should be possible after that
rjbou: slight blocker: we need to test on different platforms that we do not have CI for.
altgr noticed alpine issues too
avsm create a meta issue for manual tests?
samoht what are the fallbacks. rjbou: in the conf package CI there could be a check.
rjbou: contact the reproducible builds people to generate
avsm: can we get rid of conf files that have bad build rules?

compiler:

emillon working on the plugin to bypass compiler compilation
packages that embed their build time in their source code, and when they got last built and installed.
it didnt do what emillon expected as there is some caching. It doesnt use the source tree but there is a copy.
(this is using --in-place-build)
will be most useful when working on the build tree
no need for in place rebuild for remote branches
assume-built has changed recently - master branch currently broken but using either 2.0 or 2.1alpha2 should work.
need to be careful to replace dependencies first before using assume-built. But this is ok for the compiler plugin since.
emillon: knows the build instructions for ocaml, so can run those in ocaml source tree, then do assume-built to get this into the opam world, and then opam will

recursive pins:

need to figure out how to make this usecase work
works for companies with monorepos with unrelated packages but not so well with interlocked packages.
decision: push out post opam 2.1. Still too early in the design phase.

12 Jun 2020

Agenda

Review last week's blockers
- (none)
Spec Review
- Lock file spec
- Depext spec
- Compiler spec
- Recursive pinning spec
New specs
- user-setup moved to ocaml-opam
Switch invariants
- add a mechanism to add a version constraint to work with the compiler scheme

Notes

Present: @dra27 @rjbou @avsm @altgr @samoht @emillon. Notes by @dra27 edited by @avsm

Specs

opam-depext

@rjbou has a fix to deal with the mariadb/mysql issue which causes the solver to prefer the installed packages always
@avsm remaining issue on removing the plugin? @rjbou keep the plugin for compatibility with scripts only with a post-install message as appropriate to warn that it shouldn't be being installed any more.
@avsm No further material changes, just bug-fixes.

@avsm - spec committed.

opam-lock

Possible use-case for learn-ocaml - @AltGr looking into it further
@avsm - not clear how to use it for multi-package repositories?
Needs to warn that the locks should only be made for binaries. @AltGr flags can be used in 2.1 - need to check that 2.0 ignores them. @avsm CI use-case might want to use lockfiles. @avsm What about during rev-deps testing - lockfiles ignored at that point?
@avsm - so a clear ban on lockfiles in the repository
@avsm - will using lockfiles mean that bounds in opam files don't get updated. @AltGr CI on opam-repository should catch it - @avsm although this pushes more on to users. @dra27 Perhaps CI for packages themselves should always be testing both with and without lockfiles to ensure the bounds are correct?
@AltGr - at the moment, the lockfiles aren't used by default. If you're creating a switch just for the application, so should opam switch create . use the lockfile by default.
@avsm - perhaps not the default for now, but in future it could become the default?
@samoht - more and more packages are starting to put hard constraints in on packages; need to be careful that lockfiles don't encourage this further - it may just be that we need to add options to ignore certain constraints.
@avsm - for Jane Street 0.14 release, would be useful to be able to specify ignoring of specific constraints. Possible of "unlock" files? @samoht - this leads to not wanting to change the default workflow.
@avsm - wondering if locked is the wrong terminology (there's --lock-suffix which simply defaults to the .locked)? @emillon - could also want to use this to test against incompatible different versions (e.g. have a lockfile which builds against core 0.12, 0.13 and 0.14).
@avsm - missing for the binary use-case is a check/block that a "binary" package didn't install libraries. @AltGr - could add tag or a field. @avsm no problem with a flag as long as 2.0 just ignores it.
@avsm - for now, the arch/os problem doesn't block as the lock file can be manually updated. @samoht - could consider adding some support for being able to work out how to generate it?

Three use-cases:

learn-ocaml
irmin-unix
Jane Street core releases (unlock test)

Compiler

Plugin can create switches. It wraps opam update to follow the branch and updates the compiler. At the moment, the only 2.1 feature is the unsafe reinstallations, so it should be easy for the plugin to work in 2.0 and error only if the required feature is not present.
Possible future direction to look at using the Dune build, or a hybrid where make just builds the things not already built by the compiler's dune files.

multi-package repos

Decision - only one version of each package in each repository? It's considerably more complicated if it's permitted to encounter more than one version of a package's opam file.
Pinning and installing a subset of packages. Need to be able to ensure that if you're installing just one package, but need a way of specifying that all of that packages are pinned as well.
New field needed if the packages want to be published from a subdirectory. @rjbou - at the moment as implemented the support is only for pinning; intentionally blocked from the repository. @samoht - if there's only version of each package in a repo, then the subpath field isn't needed (because you can scan to find the package).
@avsm Fundamental problem this is trying to solve is that the opam files don't want to be in the root of the repo (Tezos being the big use-case)
Can get an example from Jane Street (mono-repo), Cohttp (virtual packages).
Does dune-project opam file generation solve these use-cases?
@avsm no matter how well opam solves this, it remains a Dune problem (in order to keep it for 2.0)
@avsm The feature still feels part-baked - feeling at the moment would be to convert cohttp to use this model and see how it plays out?

@avsm - keep the code, but move the exposing of it to 2.2 @rjbou - the feature at present this is all for pinning, not for package release.

@samoht - however, it's the case once users can pin this way that the request will follow to publish this way too.

@avsm - postpone the feature to 2.2 when the whole thing can be looked as seems better.

@AltGr - could squeeze in the recursive lookup of opam files?

@avsm - risks that then being relied on later!

Conclusion that this will be disabled for 2.1 and then picked up potentially as the headline feature for opam 2.2.

switch-invariants

@dra27 Is it worth adding a function over parts of the version?! e.g. have a function which says there's "+flambda" in the version?

@AltGr Alternative is to split ocaml-variants properly so that it becomes a virtual package with, e.g. ocaml-variants-flambda, etc.

6 Jun 2020

Agenda

Review last week's blockers
- beta checklist agreement (https://github.com/ocaml/opam/wiki/Release-2.1.0#to-beta)
- opam-lock for @AltGr (on list below to discuss)
- opam file format; who is releasing it?
Spec Review
- High level question: how to tie issues ("feature wish") to specs on the wiki
- Lock file spec
- Depext spec
- Compiler spec
- Recursive pinning spec
New specs
- https://github.com/ocaml/opam/issues/4212 - decide 2.1 story (tighten the switch invariant; change opam-repository; add a feature - relates to @dra27’s proposed poor man’s package parameters)
- lint moving to a plugin for independent releases - create issue for pending specs?
  - @rjbou Lint: Just the plugin todo, everything is present in opam
  - @rjbou: Split opam lock specs (plugin import) / opam switch freeze lock specs (new feature)
- multi-opam repo
  - @avsm started on this, will take it.
- client mirroring spec also needed for CI team.
Plugins:
- user-setup moved to ocaml-opam; https://github.com/ocaml-opam/opam-user-setup/issues/40

Notes

Present: @dra27 @rjbou @avsm @altgr @emillon @samoht. Notes by @dra27 edited by @avsm

Beta checklist looking fine
opam-file-format release still needed

Review of Specs

opam-lock

Not at consensus on the feature. Pull the feature for now? @AltGr - the support is already baked into 2.0, the UI is in the plugin and so can be pulled out. @AltGr proposed partial solution where the solver is used but the solution given is checked against the lockfile - with a warning that what has happened in the switch has deviated from the lockfile. @samoht - better than nothing, but not brilliant that there would be systemic warnings, e.g. for osx users in the irmin use-case. @avsm support is in 2.0 but it's not widely in use (?)

@samoht - can the plugin work with 2.1 if the builtin command is disabled? @AltGr - the feature is implemented in two parts. Creating lockfiles prior to 2.1 was done using the plugin; using the lockfiles has always been part of 2.0

@avsm - just commenting out the CLI integration should work?

@rjbou - two different features. The existing feature works to freeze an existing switch - the lockfile spec is new.

@avsm - uncouple viewing the CLI commands as features themselves.

@dra27/@samoht - not clear why the freezing existing switch isn't a subset of the spec - i.e. why you'd choose to freeze an existing switch if you have the ability to create general lockfiles.

@samoht - regressing the workflow of the plugin is not a problem (i.e. the fact that the plugin supported altering existing switches doesn't matter)

@AltGr - could possibly look at the incremental version. So the decisions on variables are recorded in the lockfile and on subsequent invocations of the lockfile where those variables change then the lockfile is updated with the new information.

@samoht - until the spec is finalised, can leave the feature as it is.

@AltGr - at present, lockfiles are only used when --locked passed to opam switch create. One of the considerations was whether using the lockfile should become the default in 2.1.

@avsm - come back to defining the three use-cases. One further week on the spec. @dra27 - Dune's Travis CI may well be a good candidate for using the lockfile.

opam-depext

@avsm - looking ready for committing to. Looks like (certainly on macOS) that opam update will be slower, so let's ensure that's documented. The issue on macOS is simply that brew is slow.

@AltGr - any command which needs to know what's available on the system will poll the package manager.

@dra27 - there must be OS package-manager-specific ways of detecting whether anything's changed since last time (e.g. brew must write to something if it's updated what's installed) as a fast way of determining if the full poll is required.

@AltGr - good idea, not critical for release, therefore. @avsm - indeed, as long as it's not dreadful, speed doesn't have to be a blocker.

@samoht - concern to ensure that the system packages don't get removed by proposed solutions. @AltGr - indeed, there's been an issue on the tracker with confusion over the new CLI.

Final decision on this spec in two weeks.

Compiler development

@emillon - next step is to produce a plug-in (a la ocaml-compiler-conf) which wraps the required commands for experimenting. Work on plugin has started.

@avsm - will the plug-in potentially be able to support both opam 2.0 and opam 2.1.

Issue at the moment determining the version (existing +trunk packages just download a tarball).

@dra27 - ensure that 2.1 has the required features (even hidden, or with workarounds) to ensure that plugin can do it.

@avsm - after a quick and dirty plugin to prototype the workflow with aim of ensuring what's blocking 2.1. Also good to be able to test the plugin against other repos - e.g. against multicore ocaml PRs.

Commit the support for the plugin for 2.1 (i.e. opam 2.1 should have the necessary features for the plugin to work).

@avsm to move committed specs to be a project (so that issues can be tracked against them).

Final decision on this spec in two weeks. Focus specifically on being able to handle the in-place compiler.

recursive-pin / multi-repo

@samoht - basic usecases are defined, working on refining the text.

@avsm - concerned with opam install . --recursive - issue is that you don't have control on disabling how this goes (e.g. with broken packages). @samoht - current control is too limited, might want to be able to have some kind of filter added?

@avsm - in 2.1 feature, having had opam install . pin everything, how do you unpin. @samoht - workflow doesn't necessary need this, you use it to get the dependencies and then dune build

@dra27 - concerned that this is a feature which is expected. @samoht - fairly sure that this is close enough to be fine; it's just ensuring that the work-flows are suitably refined.

Final decision on this spec in two weeks.

switch invariants

ocaml/opam#4212 needs to be checked.

@AltGr polishing the spec for next week to add patch version upgrade usecases and system compiler changes.

Other things

@dra27 hoping to pick up Cygwin/opam-repository-mingw depext things next week
@emillon looking into ocamlformat integration for opam-user-setup
@avsm fixing the gcc10 issue in opam-repo before installing the Dockerfile with a new base image.

29 May 2020

Agenda

Review last week's blockers
- Dockerfile for @avsm
- Release checklist for @rjbou
- opam-lock for @AltGr (on list below to discuss)
Spec Review
- Lock file spec
- Compiler spec
- Depext spec
- Recursive pinning spec
New specs
- lint moving to a plugin for independent releases
Platform
- @MisterDa is working on Windows compilation
- opam build: shift to 4.08 minimum and refresh build for dune build support
- Also for Windows https://github.com/ocaml/opam/pull/4137

Detailed Notes

Attendees: @samoht @rjbou @altgr @dra27 @avsm @emillon

Blockers

@AltGr @avsm - opam.ocaml.org Dockerfile nearly ready; expecting to have something ready today. Edit: in fact, the test run completed and passed during the meeting
@rjbou - only remaining tasks on the alpha->beta switch is CI; other tasks need adding? Tasks should be added on https://github.com/ocaml/opam/wiki/Release-2.1.0. @avsm specs need to be finalised - so that's the main item before we can progress to beta. @rjbou suggests adding a checklist of the specs to the 2.1.0 release page; all agree that would be very useful.

Specs

@avsm Generally note that specs are about user workflows, rather than features (so, for example, compiler pinning and switch invariants overlap in terms of implementation). @samoht notes users will care about how to install a system compiler (and that that hasn't changed) rather than that they're using the new advanced feature of switch invariants.

opam-lock

@AltGr - not sure exactly what we want to have working for 2.1.
@avsm - Architectural idea is that opam lock should do the complete solving stage and so that when the lockfile is used all that is done is evaluating variables (os, arch, etc.).
@AltGr - at present, the solver would be used during installation as normal (e.g. to identify existing packages which need to be removed). @avsm Could the lockfile encode the order of installation for example? @AltGr - actually, the external solver could be skipped, but the Dose SAT part of the solver (which checks the validity of the solution).
@avsm feature should be constrained in 2.1 to be used for a fresh switch, with relaxation lifted later. @samoht The lockfile includes the universe of required opam packages, so this should allow the switch to be installed without needing opam-repository. @rjbou - isn't this just opam switch export. @samoht - not quite, this wants to be able to be resolved for the arch, os, etc
@avsm 2 use cases for the lock - getting started on a project, so empty switch and similarly for CI.
@rjbou not clear how to generate the action graph without the solver - @avsm similarly, can't that be lifted into the lockfile. @rjbou/@AltGr - isn't this a new feature completely? Potentially similar to opam-bundle (although that similarly doesn't have the multi-os support, etc.)
@avsm - all the other suggestions are 80% of the featured workflow that users need. At present the other solutions are all experimental plugins, so now is the point to get the 100% feature correct as it moves into opam. Thats why the spec description of user workflows is so important.
@dra27 noted that being able to use the lockfile in an existing would then be the point to use the solver. The lockfile is in fact a series of locked actions rather than locked packages. @samoht - not convinced that anyone would ever want to use lockfile in an existing switch. @dra27 indeed - point is that the solver then comes in after the lockfile. Resolution: spec for 2.1 is that lockfiles should be for empty switches only; possibility for installing in existing switches bumped to 2.2 (probably to be dropped, unless a user is found!).
@avsm Issue with whether lockfiles should be in opam-repository - potentially only for packages which strictly install binaries only. @samoht - probably easier to keep it as lockfiles not in opam-repository. (@dra27 had a slight note-taking stack overflow here). @avsm move this usecase as a note for 2.2 -- it's a corner case.
@AltGr - locking the action is a lot of work on a basically new feature - is this appropriate for 2.1? Wondering whether the partial solution of checking that the lockfile works for a given set of a variables.
@avsm - not willing to have a feature in a major release which then changes in a subsequent release or doesn't match a spec. We need to release solutions to user workflows, not individual CLI features. Conclusion: @AltGr to iterate back on the implementation with the two usecases in mind, and bring it back to the dev meeting.

Compiler spec

@emillon introduced spec. @avsm on multicore, note instructions: https://github.com/ocaml-multicore/ocaml-multicore/blob/parallel_minor_gc/ocaml-variants.opam. @rjbou several PRs related to this to unbreak assume-built. This workflow is critical to the compiler dev team as well, so high priority for 2.1
Beta/alpha releases of OCaml - @dra27 noting that there wants to be some kind of ocaml-specific opt-in. @avsm - possible use of compiler flag to indicate this? These are currently special cased in the opam file format (with a compiler flag) and so it is entirely appropriate to encode special logic in there for a good user experience.
@AltGr - fairly sure that most of the required features are in-place for these use-cases. Next step is to expand the details of what the work-flows should be. So expanding the work-flows and then identifying the bugs in what's there at the moment. @rjbou - other question is using a locally-built compiler as a system compiler. @avsm noting that th version wants to match up (e.g. a multicore compiler wants to be ocaml-system.4.10.0+multicore). @AltGr at present that's definitely not easy. Distinction for the spec is using a locally installed vs locally built.

depext spec

@avsm Fourth point for 2.1-beta adding in --only-depexts or equivalent option to install only the depexts.
@avsm Also noting the deprecation of the depext plugin - so adding opam depext command which notes that it's deprecated and will go in opam 3.0. @AltGr - it's quite a lot of code to add to opam just to display that the workflow is deprecated. Wouldn't it be easier just to display the deprecation in the plugin. @dra27 users could be forced to switch with a warning that the plugin will artificially conflict with opam 3.0.
So opam depext in 2.1 can display a note advising how to access the new feature and that opam depext will cease working in opam 3.0. The opam-depext plugin can display a warning for all users (including 2.0 users) that opam-depext will not work in opam 3.0.
@rjbou - looking into mariadb/mysql issue. @AltGr clarifying that the issue is that user has MySQL installed, but opam could select conf-mariadb and then uninstall MySQL.

multi-repo spec

opam install . everything gets pinned as wanted. Missing is opam install . --deps-only which would be used, for example, to set-up dune builds but this doesn't pin properly.
@samoht fleshing out the use-cases to bring back to dev meeting next week.

opam lint

@avsm moving opam lint out to a plugin to make it easier to extend without needing an opam release.
Consensus from all that this would be useful.
@rjbou > ~~Before that add a way to add additional lints in opam lib. Main linting remains in opam~~ everything is already here for

Repository moving

@avsm moving main repos to ocaml-opam today
@AltGr to move Camelus.

Minimum OCaml version to build 2.1

No objection to 4.08.1
@avsm @dra27 to work with @misterda to get opam master branch compiling with modern dune and refreshed depexts. make cold needs to be updated to 4.10.0

22 May 2020

Report on actions progress from previous meeting

Last weeks blockers:
- @AltGr Dockerfile infrastructures for machine migration
Last weeks actions:
- @avsm deleted wrong repo label and transferred issues to opam-repository
- report on spec progress: depext, lock, ocaml compiler, subdir pin
- @rjbou to investigate ocaml-opam project board
- @AltGr to transfer camelus to ocaml-opam
Agenda:
- Review specs:
  - depext, lock, ocaml compiler, subdir pin
  - assign next specs

Notes

Attendees: @rjbou @avsm @dra27 @altgr @samoht. Notes taken by @dra27, edited by @avsm

OCaml platform and other high-level things

Springing out of the OCaml Release Readiness work, can shortly introduce a platform "meta" package in opam-repository soon which "locks" the releases of the tools.

Last week issues

@avsm moved non-opam issues to their respective repositories
Get specs to the wiki, then the issue sweep next (process for producing the specs can be advertised on Discuss after the 2.1 specs are done)
@rjbou blocked by opam-file-format 2.1 - @dra27 will actually do the PR today!
@rjbou identified depext problem with brew - some packages specify @version tag in depexts for brew and some don't; can't fix both uses at once. @avsm noting that brew allows multiple versions to be installed, and then symlinks the active one, but it's quite brittle (putting it mildly). @avsm providing a cloud mac for now, but would be better to use actual hardware (just because of expense). Problem looks as though conf-llvm is a single package, where each major version should be a separate package (to match how OS distros treat the package).
@avsm proposal to split opam-repository
- First repo is the base compilers only
- Second repo is only depexts => considerable CI runs here (can be expanded to test all the known platforms)
- Third repo is third-party packages => CI can be reduced here
- Can use opam-publish and dune-publish to create pull requests to the correct repo
- @AltGr concerned that it would break git uses of ocaml/opam-repository - @avsm clarifying that the other three are new repos and all three would be aggregated to ocaml/opam-repository so there would be no visible change to users, except those contributing compilers and plugins.
@rjbou ran a healthcheck over the repository finding bad checksums and missing archives. @avsm two parts to the fix: temporary failure of the upstream archive. Problem is client-specified mirrors (opam 2.2) - at the moment there's a mismatch as the mirrors are only specified when using the https backend (can be manually added even when using the git remote). Second part of the problem is the permanent failure of upstream - need to rewrite upstream to be the mirror. e.g. ocaml/opam-archives repository. @avsm to create repo. @AltGr noted there are hidden package mirror URLs already active.
Tangential discussion of flags - compiler and conf therefore can't be used in the package repo. plugin could similarly be moved to another repo - @avsm suggested that this could be useful to require plugins to vendor their dependencies.
@avsm to produce a spec for the split of the opam-repository

Depext spec

@samoht noting --with-test vs --depext; @rjbou/@AltGr seeing the use-case as different; @dra27 pointing out that installing a package without depexts opens the possibility of building the depexts instead. @samoht major point that the naming of a new command-line option is is important, as it must be supported for longer.
@avsm aim should be to remove the plugin completely (so adding an opam depext command directly which maintains the CLI of the old plugin but using entirely built-in features)
Pulling downstream @fdopen patches for depext into the main repo. @dra27 to check that this doesn't conflict with opam when used in Cygwin itself.

Lock spec

@avsm noting that lockfiles can break depending on the architecture on which the lockfile is generated
@samoht suggesting that the lockfile could instead be an overlay; @AltGr doesn't expect that to be too hard
@avsm need to look at being able to partially evaluate formulas so that the lockfile can be the most general locked formula. @rjbou noting that this is essentially a rewrite of the functionality. @AltGr suggesting a halfway solution where the lockfile can be incrementally extended (although more complicated).
Conclusion is that the current implementation "lock the current switch" is probably not what users expect
Another halfway house would be to restrict the variables (e.g. os) and at least warn that the lockfile might not work
Core principle for the lockfile is that you shouldn't need the solver; @avsm suggesting putting the lock in a new field: the lock is a formula, but which is never passed to the solver.
learn-ocaml and irmin as sample projects to begin with.

Compiler pinning

@emillon has produced a draft spec - @avsm working on it this week

Recursive+Subpath pinning / multi-package repo

@samoht has been working on the spec; various questions in the draft spec.
Long had the ability to have multiple packages in the same repo - the feature extends the support to make pinning more natural for repos containing multiple packages
The subpath feature aims to make monorepos easier to work with in opam. Problems identified with repos containing vendored code.
For @samoht, the recursive usecase is clear (could it even become the default) - the subpath part less so (ocaml/opam#3499 with details of the file handling for subpath pinning)

15 May 2020

Agenda

Report on actions progress from previous meeting
- @avsm: creation of ocaml-opam org to move repos into
- @avsm: creation of draft specs for opam 2.1
- @rjbou: to check for missing issues on todo list
  - best-effort ui: display ignored packages
  - add a way to add lint via api (for camelus/opam-publsh, issue already opened on the latter)
- @avsm dev reports
Decision
labels for the project (https://github.com/ocaml/opam/labels).
Discuss: security implications of https://github.com/ocaml/opam/pull/4166
- fixed in https://github.com/ocaml/opam/pull/4197
Decision: when to schedule issue cleanup and activate bot

Feature Dive:

opam-depext spec: https://github.com/ocaml/opam/wiki/Spec:-opam-depext-integration

Find owners for these specs:

opam-lock: opam team (#3734)
subdir pins: @hnrgrgr/tezos team
switch invariants: @dra27/@altgr (part of windows too)
better switch creation for local compilers: comes with switch invariant, related to #3713 @lpw25, #3708 @avsm, and #3932 @vogler

Meeting Notes

Attendees: @rjbou @avsm @dra27 @altgr @samoht. Notes taken by @dra27, edited by @avsm

@avsm creating a new GitHub org.

ocaml-opam organisation created to house opam-rt, etc. ocaml/opam and ocaml/opam-file-format staying where they are - just the dependencies.
Other projects doing similar - the idea is to allow admins to be created for individual OCaml projects as they get bigger, which can never be done for ocaml/ocaml
@avsm noting that GitHub redirects only once, so if the repos move again then links will break

Actions update

@avsm depext spec took some time; need to assign owners to specs.
@rjbou TODO list issues. Personal TODO list item is improved API to allow Camelus and opam-publish to add new lint items.
@avsm Experiment with organisation-wide project board on ocaml-opam; actioned to @rjbou
@AltGr - working on the Docker containers; nearly ready to hand to @avsm.
Release readiness team for Platform tools is forming upstream in OCaml dev team for the 4.11 release and onwards. For opam, specifically want to avoid the execvpe debacle from 4.05/4.06
2.1 release - no problem if the release takes several months to be released, but it's critical that we bring our issue backlog and design process back up to speed as it has become overwhelmed by the number of requests.

Issue and spec sweep

@avsm moving the 4 STATE: WRONG REPOSITORY issues and then deleting the label
we deleted a number of unused labels live in the meeting, which were either unused or of no historical import
@avsm proposes pausing development to deal with the specifications and backlog of issues. Consensus from team that this would be useful. Focus is on the wiki specifications first.

Security issues

@samoht suggested private project for security issues.
@avsm On specific issue with /tmp erasing. Additional guards should be in place for deleting directories which are outside the opam root.
@AltGr its unlikely to affect older opam releases due to how the call was used, but @rjbou has fix for 2.1+

depext spec

@avsm - couldn't find documentation for how to turn depext off. Missing documentation for how the entire feature works, rather than specifically how the CLI works.
@AltGr cool demo of depext in action to dev team.
@avsm noted the four-levels of subcommand to configure depext support which needs simplifying (@rjbou done - there's now opam option).
@samoht note that deprecation and removal need to be in separate releases! Doing both in one go is a recipe for user confusion.
Next steps for the spec: @rjbou to own the spec and continue it (@dra27 side-note that opam install conf-mysql; opam install mysql should not attempt to install conf-mariadb because there's an available depext which is installed already)

lock spec

@AltGr to own and start work on it - use-cases initially.

Support for local compiler testing

@gasche had a script for OPAM 1.2.2 which allowed compiler development - never worked in 2.0
@rjbou - draft blog post from the past on developer use-cases
@avsm to take this one over and post a spec.

Minimum version support

@dra27 noted that packages in opam-repository depending on opam libraries are >= 4.03; Debian's oldest is at 4.05. So moving minimum to 4.03 would be completely painless. @avsm suggests tracking Dune (so 4.08 minimum) - no problem for Debian latest which is at 4.08, would need care with opam-repository to guide other packages through it (odig, etc.).

7 May 2020

Agenda

Labelling issues:
- for 2.1.0 release; what is our convention? Is the milestone in sync?
- priority issues (e.g. from opam-repository team)
- outbound priority issues for 2.2, is there a milestone/label for this?
What known issues are missing and need to be filed (such as verbose output from solver)
What's missing from our CI matrix?
- Use FreeBSD Cirrus/Travis support?
Take decisions on outstanding issues:
- what to do about opam depext changing semantics from 2.0 to 2.1
- opam install behaviour with default lockfiles.
- opam binary installer not installing opam-installer.
- #4097 - can we bump the deps to a modern compiler and use an ocaml-secondary-compiler to build opam as with dune.
Progress reports on the 2.1.0 release checklist from everyone.
If there is time: general issue tracker sweep (defer to next meeting if not).

Meeting Notes

Attendees: @rjbou @avsm @dra27 @altgr @samoht. Notes taken by @dra27, edited by @avsm

issue closing

General move to close down issues to have a more focussed issue tracker

@avsm Need an issue triage prior to beta
prioritise internal reports from team vs platform/infra/opam-repository maintainer reports vs general public since these impact the central workflow (merging to opam-repo)
convert release checklist into issues
too many labels on the issue tracker since many are historical. What's working at the moment?
@samoht comparing with ocamlformat labelling - some labels for state/lifecycle and some for type
@avsm suggesting using GitHub Projects - comparison with https://github.com/realworldocaml/book/projects/1
Project pages can be made private, even on a public repository
@rjbou: many older issues are smaller discussions; @avsm making point as on ocaml/ocaml that the triage needs to be bounced back to the reporter.
@avsm suggested that discussion should take place and the result should then close that issue but with a new maintainer-managed issue opened to resolve what will then happen. So the original issue discusses the problem and the new issue discusses the solution.
@AltGr pointing out that closed issues are of course still searchable. @AltGr also mentioned the closing bot
@avsm agreeing that we should do this - the bot is a good idea and is working well on ocaml/ocaml and opam-repository.
@avsm suggested garbage collecting unused labels; leave the current labels as they; introduce some 2.1 labelling, e.g. a label for 2.1-performance-improvements; 2.1-switch-invariants
Key thing with the new issues is that the maintainer of the issue must keep its description current - for example, the dune file generation feature.
@dra27 pointing out positive benefits of the bot on ocaml/ocaml
@avsm 2.1 label for quality-of-life improvements
@dra27 labelling for issues in infrastructure elsewhere - ocaml/infrastructure
Actions: deploy the stale issue bot next week (@avsm)
Actions: sweep of the labels (@avsm)
Using locked issues initially - then open them for discussion later
**Action: ** @avsm will create the initial priority issues and then others chime in to edit them later

Missing issues

@avsm asking what is on todo which isn't on issue list
@rjbou a lot! Generally 2.1 things
@avsm label for priority issues
@dra27 suggesting self-assigning issues to offload todo lists
@avsm creating a milestone for 2.2.0~alpha milestone - e.g. for dune generation feature
**Action: ** @rjbou to check for missing issues on todo list and file on public tracker
Note @rjbou: there are few things in personal notes

Outstanding issues

depext

@avsm deprecate the depext plugin?
@AltGr - filtering on availability of system packages
@dra27 - what does deprecation mean in terms of supporting 2.0 repository?
@avsm - semantics differences between the two
@avsm - ability to install dependencies of the current packages only (for dune builds)
@samoht - defaults in interactive need to be the best possible defaults for users who don't really read the messages
@avsm/@rjbou - non-interactive installations
@AltGr - fear of non-interactive requests intended for CI being used by user's and then invoking package manager in sudo. @avsm recommending keeping the infrastructure that there's still a step to install or enable depexts. @samoht concern with relying on messages to warn.
@avsm - push not enable to depext plugin by default, but have clear steps to turn it on.
@AltGr - depext without sudo mode is available, but not used by default
@avsm proposing conservative solution - depext not on by default; so sudo not invoked by default in opam 2.1 but with instructions for how to enable it.
Actions: @AltGr to check precise details of defaults for next meeting

Lockfiles

Lockfiles still require solver because of os-specific packages - solver still invoked and opam-repository is still capable of breaking it therefore.
@avsm - get three projects (definitely including Irmin as it has the problematic case of multi-OS dependencies) to use lockfiles and confirm that it works.
@avsm - concern that the feature sets expectation too high (npm-style) when it clearly doesn't provide a portable lockfile.
@AltGr - also uses lockfiles on a shell-script deployment system he runs using opam, so thats another candidate
@avsm - partially evaluate the solver?
@AltGr - could possibly increase the detail of the lockfile as it runs on different architectures
@avsm - use cases: irmin binary and @AltGr's server deployment

binary installer not installing opam-installer

install.sh only installs opam, not opam-installer - @avsm suggesting integrating the opam-installer command as a command within opam itself. @AltGr suggesting busybox style for opam-installer as a symlink.
Actions: @avsm to create an issue

Base language support

@dra27 not initially convinced of premise of the issue
Possible switching of minimum version to something more recent, perhaps match dune. This only affects the binary and not the supported switch ranges.
Reasoning is to lower our test matrix since we rarely test new releases built with (e.g.) 4.02.3 and distros have moved on.
@avsm can modernise codebase and use newer OCaml features. @Altgr likes this.
@samoht defer the change until after 2.1 release?
Warning-free build wanted using modern opam
Switching repo files to opam 1.2 format is overdue now
Actions: All to do research and take decision on the next dev meeting.

29 Apr 2020

Present: rjbou, altgr, samoht, dra27, avsm, thomasblanc

Agenda:

Resuming regular opam development meetings. Goal is to document and mechanise the opam 2.1 release process, with the aim of reducing time between releases for future releases.
Meetings will be regular, initially with a strong focus on release management for opam 2.1.0.
After that we will use the meetings for roadmap and planning of future releases.
Went through release planning for 2.1, resulting in wiki entries for a general release cycle and opam 2.1 release cycle checklist specifically.
Did not get a time to go through individual features in 2.1. Deferring those for a future meeting.

Other actions:

@avsm to create VM for opam staging server and give @AltGr access.
@avsm to create an ocaml-opam/ org on GitHub to store project related repos (e.g. Camelus).