GitLab CI for Windows & OSX

[saraedum]

Developers currently don't have an easy way to test proposed changes on OSX & Windows if they do not have access to such machines themselves. Given the flakiness of the patchbot on Windows, it would be nice to provision GitLab CI infrastructure for these systems.

We can not use the docker setup unfortunately, as docker on Windows is not ready to run fork-heavy workloads (#25805) and docker on OSX does simply not exist.

The current idea would be to provide SSH runners on Linux machines tagged as osx & nt that run QEMU/KVM. An incoming CI request would then take an LVM snapshot of an existing Windows/OSX machine, start a QEMU/KVM machine on it, and run its CI via SSH.

CI on the "develop" branch would probably start from a clean OS snapshot and tag the resulting LVM volume to "develop" so the build artifacts can be reused by runs on other branches.

GitLab does not support libvirt officially but there is this runner that seems quite reasonable as a starting point.

Performance

OSX

Sage builds from scratch on QEMU on a dual core with 2GB RAM and a seeded ccache in 160 minutes. With local/ intact from a previous build it rebuilds in seconds.

Windows

Sage builds from scratch on QEMU on a single core with 1.5GB RAM and a seeded ccache in about 500 minutes.

real    472m37.961s
user    284m29.903s
sys     180m23.735s

With local/ intact from a previous build it rebuilds in seconds.

Legalities

OSX

Relevant seems to be 2.B of their license in particular

to install, use and run up to two (2) additional copies or instances of the Apple Software
within virtual operating system environments on each Mac Computer you own or control that is
already running the Apple Software, for purposes of: (a) software development; (b) testing during
software development; (c) using macOS Server; or (d) personal, non-commercial use.

Now, the "already running" bit is weird because that means that we cannot use Linux as the base system?

This bit is the same for all the languages that I can read except for the German one which surprisingly omits the "already running":

Zum Installieren, Nutzen und Ausführen von bis zu zwei (2) zusätzlichen Kopien oder
Exemplaren der Apple-Software innerhalb virtueller Betriebssystemumgebungen auf jedem Mac-
Computer, dessen Eigentümer du bist oder der deiner Kontrolle unterliegt, zum Zwecke der: (a)
Softwareentwicklung; (b) Durchführung von Tests während der Softwareentwicklung; (c) Nutzung
von macOS Server; oder (d) der persönlichen, nicht kommerziellen Nutzung.

So, it's Ok to run two instances of OSX virtualized on Apple Hardware. I could imagine that it would be Ok to run more than two if we used different versions of OSX and therefore got separate licenses but I am not an expert here.

Windows

Can we use the trial versions of Win10 for testing purposes? I guess not, but we should check.

Depends on #24655

CC: @embray @slel

Component: build

Keywords: osx, windows, ContinuousIntegration

Issue created by migration from https://trac.sagemath.org/ticket/25980

[saraedum]

Dependencies: #24655

[embray]

comment:4

I have no idea about the legal issues w.r.t. using Windows VM snapshots for testing. I have a Windows 10 image I created for my own Windows VMs that comes installed with a valid(!) license provided by CNRS for all Windows 10 installs, so I don't know what the limits are, if any, to spinning up VMs that use that license. In any case I can easily set up a snapshot that already has Cygwin and SSH pre-configured.

[saraedum]

comment:6

I did not manage to build on cygwin in one go due to a [combinatorial_designs-20140630] Waiting for rebase lock where it hung forever. Now that I have a primed ccache I'll try again and see how it goes.

I could not get High Sierra to work. I had manged to install it once but it was extremely slow. I am now trying with Sierra which seems much better so far.

[saraedum]

comment:7

Sage builds on Sierra running on QEMU on a Linux box. The build took 270 minutes on a dual core with 2GB RAM which sounds quite reasonable to me.

[saraedum]

comment:9

embray: Would you mind to comment on the timings in the ticket description? Is that roughly what you see on a native Windows system?

[embray]

comment:10

FWIW from my recent build from scratch for the 8.3 release (this is likely with some ccache seeding but probably not for everything):

real    332m31.190s
user    515m41.689s
sys 249m16.470s

This is with SAGE_NUM_THREADS=8.

[embray]

comment:11

Replying to @saraedum:

I did not manage to build on cygwin in one go due to a [combinatorial_designs-20140630] Waiting for rebase lock where it hung forever. Now that I have a primed ccache I'll try again and see how it goes.

This is a bug I've seen sometimes but am not sure how to reproduce, where in fact there are still other packages building in the background, but make doesn't switch which job it's following to print to stdout so it looks like it's hanging, while in fact other jobs are still running.

[saraedum]

comment:12

On my single core Windows virtual machine I get for sage -t --long --all

----------------------------------------------------------------------
sage -t --long src/sage/all.py  # 1 doctest failed
sage -t --long src/sage/combinat/matrices/dancing_links.pyx  # 17 doctests failed
sage -t --long src/sage/interfaces/gap.py  # Killed due to kill signal
sage -t --long src/sage/misc/sagedoc.py  # 7 doctests failed
sage -t --long src/sage/tests/cmdline.py  # 1 doctest failed
sage -t --long src/doc/common/conf.py  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 22189.5 seconds
    cpu time: 16212.7 seconds
    cumulative wall time: 20648.1 seconds

real    370m12.148s
user    260m5.826s
sys     82m50.626s

[saraedum]

comment:13

embray, how does that compare to native? In any case, it seems very reasonable to me. If we would run eight of these somewhere that could be quite valuable.

[embray]

comment:14

Well, I'm running make ptestlong for 8.4.beta1 right now on my laptop (I have designated SAGE_NUM_THREADS=4 for now) so I'll see what that says.

There are a number of tests--the ones you reported as failed being some of the common ones (such as src/sage/tests/cmdline.py) that are slow and fail often due to timeout. Re-running them, or increasing the timeout, usually fixes it.

[saraedum]

Changed keywords from osx, windows, CI to osx, windows, ContinuousIntegration

[mkoeppe]

comment:16

outdated