Setting up MMDevice and MMCore mirror repositories

[marktsuchida]

I want to continue along the plan set forth in #418, but there is a chicken-and-egg problem that needs to be solved:

• We want to keep all code in mmCoreAndDevices where it is, for as long as possible, with the existing build systems working, as we work on the migration.

• But it is hard to have proper Meson-based builds of pymmcore and MMCoreJ unless MMCore can be consumed as a unit (and likewise for MMCore, we need MMDevice as a unit). This is much cleaner if MMDevice and MMCore are each available in their own repositories.

  • The reason for this is that Meson needs to put subprojects (dependencies that are built together) into a flat subprojects/ directory (a limitation that has good reasons behind it). Because MMCore depends on MMDevice, we need to have subprojects/MMDevice and subprojects/MMCore when building pymmcore, but the MMCore Meson build needs subprojects/MMDevice in it's tree, too.
  • Subprojects will be fetched using Meson "wraps" (i.e., not Git submodules), which cannot extract subdirectories of a repository.
  • For pymmcore, there is also a specific issue with using the whole mmCoreAndDevices as a Git submodule, which is that there is no good way to build a correct source distribution.

We can move forward with these things by having mirror repositories (micro-manager/MMDevice and micro-manager/MMCore) that can be used to build MMCoreJ and pymmcore. Here is the idea:

• The 2 mirror repositories are strict mirrors of the corresponding subdirectories of mmCoreAndDevices
  • There will be automatic sync that uses git-filter-repo
• The official code remains in mmCoreAndDevices; the mirror repos do not accept pull requests
• The mirror repos will be useful only when using the experimental Meson build scripts within them
• The mirror repos will not run their own CI

Once this is available, I envision the following setup for the next stage:

• The MMDevice Meson build will be tested in GitHub Actions as it is currently (in mmCoreAndDevices)
• The MMCore Meson build will be tested similarly, but by fetching MMDevice from the mirror repo using a wrap
• pymmcore will be able to merge (an updated version of) Meson build, replacing setuptools pymmcore#109
  • And we can eliminate the Git submodule, letting the wrap specify the MMCore commit to use
  • This will also facilitate migrating to the nanobind-based wrappers developed by @tlambert03, currently in pymmcore-nano
• We can start working on a Meson build of MMCoreJ (which is somewhat analogous to pymmcore)
  • Once this is working, we can extract MMCoreJ into its own repo, have automated builds for it, and publish to Maven Central, which in turn allows us to start fully separating the Java and C++ builds.
• Conversion of device adapters to Meson builds will also be helped by being able to fetch just MMDevice using a wrap.

---

Additional considerations for the mirroring:

• Approximately we just need the result of git filter-repo --subdirectory-filter MMDevice (and likewise for MMCore).
  • Of course the commit hashes in the subdirectory mirrors will be unrelated to the hashes of corresponding commits in mmCoreAndDevices.
• But I want to take this opportunity to eliminate the inconvenient two-root arrangement that mmCoreAndDevices has:
  • The primary root commit is 3b69e76 (Feb 2007)
  • There is a second root, 4ff8efd (Feb 2021), for no good reason that I can see
  • A second copy of the source is "added back" in a6fd737 (immediate child of the second root), labeled "initial commit", which is always confusing (and distracting) when viewing history
  • That second root (after a bunch of changes) is eventually merged into the original history at 17b988c
  • It's too late to change anything in mmCoreAndDevices, of course, but I think it should be possible to linearize this part of the history so that we start from a single root, before extracting subdirectories such as MMDevice, so that the mirrored repos will not have this two-root problem.
  • Probably by replacing the first 2 commits of the second root with a branch off an appropriate commit along the first root, but TBD.

[marktsuchida]

Does anybody have a strong opinion about the repo names for MMCore and MMDevice? I was thinking it might be less confusing if we use lowercase mmcore and mmdevice for the repo names, so that we have a way of distinguishing the mmCoreAndDevices subdirectories from the independent-repo (mirror for now) versions. (Aside from all-lowercase being more common/trendy for repo names in general.)

MMCore depends on MMDevice, so we will have

• In mmCoreAndDevices, MMCore/subprojects/mmdevice.wrap (which points to the micro-manager/mmdevice repo)
• In the mmcore repo, subprojects/mmdevice.wrap (same file mirrored)
• In mmCoreAndDevices, after setting up MMCore's Meson build, MMCore/subprojects/mmdevice/ (either fetched from the micro-manager/mmdevice mirror repo or copied from MMDevice/ in mmCoreAndDevices)
  • (Note that Meson will ignore the .wrap if the subproject is copied in before configuring.)

The argument either way is not very strong, though.

Also for mmcorej vs MMCoreJ.