Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local), add homebrew packages #29104

mkoeppe · 2020-01-29T15:48:19Z

#29053 adds tox.ini to support testing the sage distribution in isolated environments via docker.

This ticket adds toxenvs local-homebrew-macos-{minimal,standard} for testing with (best-effort) isolated installations of homebrew on macOS (not using docker).

To test:

    brew install tox
    tox -e local-homebrew-macos-standard

See comments in tox.ini for more examples.

An example run of that is here: https://github.com/mkoeppe/sage/actions/runs/32812271

Depends on #29335
Depends on #29361

CC: @dimpase @jhpalmieri @mwageringel @vbraun @tscrim

Component: porting

Author: Matthias Koeppe

Branch/Commit: dfce7d4

Reviewer: Dima Pasechnik, John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/29104

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-01-29T19:02:11Z

Branch: u/mkoeppe/tox-homebrew

mkoeppe · 2020-01-29T19:05:00Z

Last 10 new commits:

`7e59a6f`	`build/bin/write-dockerfile.sh: Add commands to activate conda env`
`2ad9a97`	`Merge tag '9.1.beta2' into t/29053/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time`
`9fca230`	`add doc`
`f6cbb8a`	`add centos-{7,8}`
`c651058`	`on centos, IGNORE_MISSING_SYSTEM_PACKAGES=yes`
`98edff6`	`Add build/pkgs/gfortran/distros/fedora.txt`
`843ca76`	`build/pkgs/fedora.txt: Add python3`
`bd6a545`	`tox.ini: Add debian stretch, put sid last`
`23c6334`	`build/pkgs/openblas/distros/arch.txt: Add lapack, cblas`
`fca6654`	`tox.ini: Add toxenvs local-homebrew-macos-minimal, local-homebrew-macos-standard`

mkoeppe · 2020-01-29T19:05:00Z

Commit: fca6654

sagetrac-git · 2020-01-29T20:01:49Z

Changed commit from fca6654 to e0e8a4c

sagetrac-git · 2020-01-29T20:01:49Z

Branch pushed to git repo; I updated commit sha1. New commits:

`e0e8a4c`	`tox.ini [local-homebrew]: Fixup PATH, PKG_CONFIG_PATH, set LDFLAGS, CPPFLAGS; [local-direct]: New`

sagetrac-git · 2020-01-29T20:11:45Z

Changed commit from e0e8a4c to 16ca88f

sagetrac-git · 2020-01-29T20:11:45Z

Branch pushed to git repo; I updated commit sha1. New commits:

`16ca88f`	`tox.ini: Use {envlogdir}; use ln -sf`

sagetrac-git · 2020-01-29T21:15:44Z

Changed commit from 16ca88f to 6a1803d

sagetrac-git · 2020-01-29T21:15:44Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6a1803d`	`tox.ini [local]: First build the base-toolchain`

sagetrac-git · 2020-01-30T04:33:14Z

Changed commit from 6a1803d to 4b72987

sagetrac-git · 2020-01-30T04:33:14Z

Branch pushed to git repo; I updated commit sha1. New commits:

`4b72987`	`tox.ini [local-homebrew]: Set PATH earlier and set ACLOCAL_PATH for bootstrapping`

mkoeppe · 2020-01-30T04:35:23Z

Author: Matthias Koeppe

dimpase · 2020-02-05T17:52:49Z

comment:10

Is this ticket MacOS-specific?

mkoeppe · 2020-02-05T18:32:28Z

comment:11

Hmm, I haven’t tried linuxbrew. Might be worth checking

dimpase · 2020-02-05T18:34:37Z

comment:12

I'll fire up my vintage OSX 10.13 macbookair...

sagetrac-git · 2020-02-06T14:26:38Z

Changed commit from 4b72987 to 2141b75

sagetrac-git · 2020-02-06T14:26:38Z

Branch pushed to git repo; I updated commit sha1. New commits:

`2141b75`	`Merge tag '9.1.beta3' into t/29104/tox-homebrew`

dimpase · 2020-02-07T21:44:24Z

comment:14

I tried this:

tox -p auto -e local-homebrew-macos-standard

and got

...
[sagelib-9.1.beta2] user	0m10.244s
[sagelib-9.1.beta2] sys	0m3.812s

real	0m16.288s
user	0m10.956s
sys	0m4.047s
Sage build/upgrade complete!
✔ OK local-homebrew-macos-standard in 1 minute, 14.362 seconds
_________________________________________________________________________________________________ summary __________________________________________________________________________________________________
  local-homebrew-macos-standard: commands succeeded
  congratulations :)

not tests were run, it seems, is this normal?

mkoeppe · 2020-02-08T22:44:54Z

comment:15

The default packages set in tox.ini are only fflas_ffpack scipy r, and those (and their dependencies) are checked.
I don't know why it's actually building sagelib, but I hope to fix bizarre build behavior like this in #29113.

mkoeppe · 2020-02-10T00:12:02Z

Changed dependencies from #29053 to none

sagetrac-git · 2020-02-16T20:44:12Z

Changed commit from 2141b75 to 4eb201b

sagetrac-git · 2020-02-16T20:44:12Z

Branch pushed to git repo; I updated commit sha1. New commits:

`4eb201b`	`Merge tag '9.1.beta4' into t/29104/tox-homebrew`

mkoeppe · 2020-03-20T02:58:19Z

comment:118

Replying to @mkoeppe:

Replying to @jhpalmieri:

Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?

Good idea. WIll do

I'll do this in #29146

mkoeppe · 2020-03-20T03:09:18Z

comment:119

Replying to @mkoeppe:

Let's fix the FLINT issue on a follow-up ticket.

This issue is probably the same as #27708/#28409.

mkoeppe · 2020-03-20T03:09:56Z

comment:120

Ready for review

dimpase · 2020-03-20T04:26:04Z

comment:121

Replying to @mkoeppe:

Replying to @dimpase:

I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.

Well, for the build I'm using high parallelization on purpose in order to catch possible parallelization bugs in the scripts.

One can split building into make built and make ptest, with the latter, much more memory hungry, run with lower value of -j.

sagetrac-git · 2020-03-20T16:01:54Z

Branch pushed to git repo; I updated commit sha1. New commits:

`dfce7d4`	`tox.ini [local]: Pass MAKE environment variable, do not default to 'make -j12', change default target to 'build', add doc`

sagetrac-git · 2020-03-20T16:01:54Z

Changed commit from 467160c to dfce7d4

mkoeppe · 2020-03-20T16:02:31Z

comment:123

OK I've made it more configurable already on this ticket.
And I have added documentation

jhpalmieri · 2020-03-20T18:40:47Z

comment:125

@mwageringel, @dimpase: do you think more needs to be done here, or can we proceed with a positive review? I'm happy with it.

dimpase · 2020-03-20T18:48:47Z

comment:126

I'm 8 timezones away from my MacOS machine.

I'd like more details/pointers on how the Github Actions testing is done - for this looks to me like "all sufficiently advanced technology is indistinguishable from magic" kind of thing :-)

mkoeppe · 2020-03-20T19:12:48Z

comment:127

This ticket does no GitHub Actions testing. This is all introduced in #29087.

dimpase · 2020-03-20T19:26:41Z

comment:128

how does one get the data I linked in comment:126 then?

mkoeppe · 2020-03-20T19:29:05Z

comment:129

By merging #29087 and pushing the branch to a GitHub repository.

mkoeppe · 2020-03-20T19:30:27Z

comment:130

If the GitHub Actions interface is overwhelming, just download the "Artifacts". They are tar.gz files of build logs.

dimpase · 2020-03-20T19:35:29Z

comment:131

Replying to @mkoeppe:

By merging #29087 and pushing the branch to a GitHub repository.

yes,but that's how one triggers this - I'm asking rather about how stuff works, inside. Locally and then somehow pushing the logs to GitHub?

mkoeppe · 2020-03-20T19:42:52Z

comment:132

I don't push logs to GitHub.

#29087 adds .github/workflows/tox.yml (https://github.com/sagemath/sagetrac-mirror/blob/576a18834a16ff05cd6894172a55e86bcc95d407/.github/workflows/tox.yml&id2=be1e22c441d9332fd0bf4a3066e1964deddaf8c7) which gets run on every commit pushed to GitHub and on every pull request.

The job tox-local-homebrew-macos defined in that file provisions a macOS host, checks out the branch and runs tox -e local-homebrew-macos-standard there. Then it packages up logs and makes them available as an artifact.

dimpase · 2020-03-20T19:52:29Z

comment:133

and where is that MacOS host getting provisioned?

mkoeppe · 2020-03-20T19:54:01Z

comment:134

GitHub takes care of that, I think it's using Azure.

mwageringel · 2020-03-20T22:05:26Z

comment:135

Replying to @jhpalmieri:

@mwageringel, @dimpase: do you think more needs to be done here, or can we proceed with a positive review? I'm happy with it.

I have not tested this locally, but as you have done that, I am happy with this too.

One last question though: As far as I can see, readline is not installed into /usr/local. Why do you not run into conflicts with macOS's system "readline"/libedit, as in #29000?

mkoeppe · 2020-03-20T22:20:27Z

comment:136

These tests use their own homebrew installation in a prefix .tox/local-homebrew-macos-standard/homebrew/, and sets up environment variables using the new script .homebrew-build-env. This sets LDFLAGS and CPPFLAGS for the readline installed into opt under the homebrew prefix.

mwageringel · 2020-03-20T22:31:02Z

comment:137

The problem in #29000 occured despite passing LDFLAGS and CPPFLAGS to configure, so I guess setting these also during make would avoid that problem, as a workaround.

mkoeppe · 2020-03-20T22:33:32Z

comment:138

I think the problem description of #29000 needs some more precision.

dimpase · 2020-03-21T02:10:15Z

comment:139

OK, looks good to me.

dimpase · 2020-03-21T02:10:15Z

Reviewer: Dima Pasechnik, John Palmieri

mkoeppe · 2020-03-21T02:16:16Z

comment:140

Thanks everyone!

vbraun · 2020-03-29T00:24:12Z

Changed branch from u/mkoeppe/tox-homebrew to dfce7d4

mkoeppe added this to the sage-9.1 milestone Jan 29, 2020

mkoeppe added c: porting labels Jan 29, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Add to tox.ini a homebrew toxenv (with a fresh install not in /usr/local)~~ Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local) Jan 29, 2020

This comment has been minimized.

Sign in to view

mkoeppe added the s: needs review label Jan 30, 2020

This comment has been minimized.

Sign in to view

dimpase added s: positive review and removed s: needs review labels Mar 21, 2020

vbraun removed the s: positive review label Mar 29, 2020

vbraun closed this as completed in 1f44f22 Mar 29, 2020

Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local), add homebrew packages #29104

Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local), add homebrew packages #29104

Comments

mkoeppe commented Jan 29, 2020

mkoeppe commented Jan 29, 2020

mkoeppe commented Jan 29, 2020

mkoeppe commented Jan 29, 2020

This comment has been minimized.

sagetrac-git mannequin commented Jan 29, 2020

sagetrac-git mannequin commented Jan 29, 2020

sagetrac-git mannequin commented Jan 29, 2020

sagetrac-git mannequin commented Jan 29, 2020

sagetrac-git mannequin commented Jan 29, 2020

sagetrac-git mannequin commented Jan 29, 2020

sagetrac-git mannequin commented Jan 30, 2020

sagetrac-git mannequin commented Jan 30, 2020

This comment has been minimized.

mkoeppe commented Jan 30, 2020

dimpase commented Feb 5, 2020

mkoeppe commented Feb 5, 2020

dimpase commented Feb 5, 2020

sagetrac-git mannequin commented Feb 6, 2020

sagetrac-git mannequin commented Feb 6, 2020

dimpase commented Feb 7, 2020

mkoeppe commented Feb 8, 2020

mkoeppe commented Feb 10, 2020

sagetrac-git mannequin commented Feb 16, 2020

sagetrac-git mannequin commented Feb 16, 2020

mkoeppe commented Mar 20, 2020

mkoeppe commented Mar 20, 2020

mkoeppe commented Mar 20, 2020

dimpase commented Mar 20, 2020

sagetrac-git mannequin commented Mar 20, 2020

sagetrac-git mannequin commented Mar 20, 2020

mkoeppe commented Mar 20, 2020

This comment has been minimized.

jhpalmieri commented Mar 20, 2020

dimpase commented Mar 20, 2020

mkoeppe commented Mar 20, 2020

dimpase commented Mar 20, 2020

mkoeppe commented Mar 20, 2020

mkoeppe commented Mar 20, 2020

dimpase commented Mar 20, 2020

mkoeppe commented Mar 20, 2020

dimpase commented Mar 20, 2020

mkoeppe commented Mar 20, 2020

mwageringel commented Mar 20, 2020

mkoeppe commented Mar 20, 2020

mwageringel commented Mar 20, 2020

mkoeppe commented Mar 20, 2020

dimpase commented Mar 21, 2020

dimpase commented Mar 21, 2020

mkoeppe commented Mar 21, 2020

vbraun commented Mar 29, 2020