Refactor sage_conf #33295

mkoeppe · 2022-02-05T17:51:34Z

(split out from #31396)

We refactor sage_conf to make it easier to maintain variants of it, both the 2 in-tree variants (and another one to be added in #31396) and downstream variants:'''

The configure-generated configuration file is now pkgs/sage-conf/_sage_conf/_conf.py
The file pkgs/sage-conf/sage_conf.py is now static

CC: @kiwifb @antonio-rojas @tornaria

Component: scripts

Author: Matthias Koeppe

Branch/Commit: eeddb03

Reviewer: François Bissey

Issue created by migration from https://trac.sagemath.org/ticket/33295

The text was updated successfully, but these errors were encountered:

mkoeppe · 2022-02-05T17:59:59Z

Branch: u/mkoeppe/refactor_sage_conf

mkoeppe · 2022-02-05T18:01:59Z

Author: Matthias Koeppe

mkoeppe · 2022-02-05T18:01:59Z

New commits:

`70c2e95`	`pkgs/sage-conf*: Refactor`

mkoeppe · 2022-02-05T18:01:59Z

Commit: 70c2e95

kiwifb · 2022-02-05T20:23:07Z

comment:3

Three version of sage_conf? So

"regular"
pypi
relocatable

when is each one used or intended to be used? Ideally, I want something from pypi or some other repo that will give me a source tarball of the package (and only that one package).

mkoeppe · 2022-02-05T20:44:45Z

comment:4

Replying to @kiwifb:

when is each one used or intended to be used?

Explained in pkgs/sage-conf/README.rst

sagetrac-git · 2022-05-03T01:20:18Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ffb5344`	`Merge tag '9.6.rc3' into t/33295/refactor_sage_conf`

sagetrac-git · 2022-05-03T01:20:18Z

Changed commit from 70c2e95 to ffb5344

dimpase · 2022-06-17T11:01:36Z

comment:7

rebase is needed

sagetrac-git · 2022-06-17T18:45:18Z

Branch pushed to git repo; I updated commit sha1. New commits:

`fbb318d`	`Merge tag '9.7.beta2' into t/33295/refactor_sage_conf`

sagetrac-git · 2022-06-17T18:45:18Z

Changed commit from ffb5344 to fbb318d

kiwifb · 2022-06-21T00:39:30Z

comment:9

I very much want this ticket in 9.7 and it looks in good shape now.

kiwifb · 2022-06-21T00:39:30Z

Reviewer: François Bissey

kiwifb · 2022-06-21T00:59:05Z

comment:10

I have spoken too soon. I see there is a file pkgs/sage_conf/sage_conf.py.in and its link in sage_conf_pypi that may still need removal. I believe it is functionally fully replaced by _sage_conf/conf.py.in.

Similarly, we have pkgs/sage_conf/setup.cfg and pkgs/sage_conf/setup.cfg.in, and the .in is not used anymore by configure either

diff --git a/configure.ac b/configure.ac
index eca011d..09f694d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -514,7 +514,7 @@ dnl AC_CONFIG_HEADERS([config.h])
 AC_CONFIG_FILES([build/make/Makefile-auto build/make/Makefile])
 AC_CONFIG_FILES([src/bin/sage-env-config src/bin/sage-src-env-config build/bin/sage-build-env-config])
 
-AC_CONFIG_FILES([pkgs/sage-conf/sage_conf.py pkgs/sage-conf/setup.cfg])
+AC_CONFIG_FILES([pkgs/sage-conf/_sage_conf/_conf.py])
 
 dnl Create basic directories needed for Sage
 AC_CONFIG_COMMANDS(mkdirs,

Once those are dealt with, I am OK with putting things straight back to positive review.

mkoeppe · 2022-06-25T18:29:33Z

comment:11

Thanks for catching this. Looks like I was not careful enough in one of the merges.

sagetrac-git · 2022-06-25T18:31:16Z

Branch pushed to git repo; I updated commit sha1. New commits:

`93ca1f7`	`Merge tag '9.7.beta3' into t/33295/refactor_sage_conf`
`9d10055`	`pkgs/sage-conf*: Remove some obsolete files`

sagetrac-git · 2022-06-25T18:31:16Z

Changed commit from fbb318d to 9d10055

mkoeppe · 2022-06-25T20:25:16Z

comment:15

Thanks!

GMS103 · 2022-06-26T21:00:57Z

comment:16

I am unable to merge this ticket on top of SageMath 9.7.beta3:

% rm -fr sage
% git clone -c core.symlinks=true --branch develop --tags https://github.com/sagemath/sage.git
% cd sage
% git-trac-merge 33295
% make configure
% source .homebrew-build-env
% ./configure

gives

config.status: error: cannot find input file: `pkgs/sage-conf/sage_conf.py.in'

(log attached).

GMS103 · 2022-06-26T21:01:31Z

Attachment: config.log

kiwifb · 2022-06-26T21:16:43Z

comment:17

OK, this is strange. It definitely shouldn't happen. Result of

grep sage_conf.py configure*

please. I mean it like it is, don't add an .in anywhere.

GMS103 · 2022-06-26T21:20:42Z

comment:18

% grep sage_conf.py configure*
configure:ac_config_files="$ac_config_files pkgs/sage-conf/sage_conf.py pkgs/sage-conf/setup.cfg"
configure:    "pkgs/sage-conf/sage_conf.py") CONFIG_FILES="$CONFIG_FILES pkgs/sage-conf/sage_conf.py" ;;
%

But I must add that this happens on macOS 12.4.
On macOS 11.6.7 it works.

kiwifb · 2022-06-26T21:24:50Z

comment:19

Somehow the branch hasn't applied, those lines shouldn't be there.

GMS103 · 2022-06-26T21:54:22Z

comment:21

In fact the problem is the following.

./configure works, but ./configure --enable-download-from-upstream-url fails as indicated.

(I was merging several tickets, some needing the option.)

kiwifb · 2022-06-26T21:56:58Z

comment:22

Replying to @GMS103:

In fact the problem is the following.

./configure works, but ./configure --enable-download-from-upstream-url fails as indicated.

(I was merging several tickets, some needing the option.)

And it should not whatever the configuration option. The lines I asked you to grep earlier should be absent if you are really on this branch.

GMS103 · 2022-06-26T22:44:41Z

comment:23

Thanks for bearing with my ignorance.

This is what happens for me every time on macOS 12.4 (but not on macOS 11.6.7).

% rm -fr sage
% git clone -c core.symlinks=true --branch develop --tags https://github.com/sagemath/sage.git
[...]
% cd sage
% git-trac-merge 33295
#33295
remote branch: u/mkoeppe/refactor_sage_conf

From git://trac.sagemath.org/sage
 * branch                  u/mkoeppe/refactor_sage_conf -> FETCH_HEAD
 * [new branch]            u/mkoeppe/refactor_sage_conf -> trac/u/mkoeppe/refactor_sage_conf

Removing pkgs/sage-conf_pypi/sage_conf.py.in
Merge made by the 'recursive' strategy.
 Makefile                                                   |  2 +-
 build/pkgs/sage_conf/dependencies                          |  2 +-
 configure.ac                                               |  2 +-
 pkgs/sage-conf/.gitignore                                  |  4 +++-
 pkgs/sage-conf/_sage_conf/__main__.py                      | 22 ++++++++++++++++++++++
 pkgs/sage-conf/{sage_conf.py.in => _sage_conf/_conf.py.in} | 25 -------------------------
 pkgs/sage-conf/sage_conf.py                                |  2 ++
 pkgs/sage-conf/{setup.cfg.in => setup.cfg}                 |  9 +++++----
 pkgs/sage-conf_pypi/.gitignore                             |  3 ++-
 pkgs/sage-conf_pypi/MANIFEST.in                            |  5 +++--
 pkgs/sage-conf_pypi/_sage_conf                             |  1 +
 pkgs/sage-conf_pypi/sage_conf.py                           |  1 +
 pkgs/sage-conf_pypi/sage_conf.py.in                        |  1 -
 pkgs/sage-conf_pypi/setup.cfg                              | 17 +----------------
 pkgs/sage-conf_pypi/setup.py                               | 10 ++--------
 src/bin/sage-env-config.in                                 |  2 +-
 src/bin/sage-src-env-config.in                             |  2 +-
 17 files changed, 47 insertions(+), 63 deletions(-)
 create mode 100644 pkgs/sage-conf/_sage_conf/__main__.py
 rename pkgs/sage-conf/{sage_conf.py.in => _sage_conf/_conf.py.in} (75%)
 create mode 100644 pkgs/sage-conf/sage_conf.py
 rename pkgs/sage-conf/{setup.cfg.in => setup.cfg} (81%)
 create mode 120000 pkgs/sage-conf_pypi/_sage_conf
 create mode 120000 pkgs/sage-conf_pypi/sage_conf.py
 delete mode 120000 pkgs/sage-conf_pypi/sage_conf.py.in
 mode change 100644 => 120000 pkgs/sage-conf_pypi/setup.cfg
% grep sage_conf.py configure*
% make configure
./bootstrap -d
rm -rf config configure build/make/Makefile-auto.in
rm -f src/doc/en/installation/*.txt
rm -rf src/doc/en/reference/spkg/*.rst
rm -f environment.yml
rm -f src/environment.yml
rm -f src/environment-dev.yml
rm -f environment-optional.yml
rm -f src/environment-optional.yml
rm -f src/Pipfile
rm -f src/pyproject.toml
rm -f src/requirements.txt
rm -f src/setup.cfg
bootstrap:83: installing 'm4/sage_spkg_configures.m4'
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_objects/src/pyproject.toml
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_objects/src/requirements.txt
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_objects/src/setup.cfg
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_environment/src/pyproject.toml
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_environment/src/requirements.txt
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_environment/src/setup.cfg
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_repl/src/pyproject.toml
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_repl/src/requirements.txt
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_repl/src/setup.cfg
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_categories/src/MANIFEST.in
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_categories/src/pyproject.toml
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_categories/src/requirements.txt
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagemath_categories/src/setup.cfg
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagelib/src/Pipfile-dist
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagelib/src/Pipfile
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagelib/src/pyproject.toml
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagelib/src/requirements.txt
./bootstrap: installing /Users/sagemath/SageMath/Git/sage/build/pkgs/sagelib/src/setup.cfg
./bootstrap-conda:48: generate conda enviroment files
src/doc/bootstrap:68: installing src/doc/en/installation/arch*.txt
src/doc/bootstrap:68: installing src/doc/en/installation/debian*.txt
src/doc/bootstrap:68: installing src/doc/en/installation/fedora*.txt
src/doc/bootstrap:68: installing src/doc/en/installation/cygwin*.txt
src/doc/bootstrap:68: installing src/doc/en/installation/homebrew*.txt
src/doc/bootstrap:78: installing src/doc/en/reference/spkg/*.rst
bootstrap:74: installing 'config/config.rpath'
./bootstrap: line 164: aclocal: command not found
Bootstrap failed, downloading required files instead.
Attempting to download package configure-b466c8ae02c04d287d28d2d669e1d111df3724ff.tar.gz from mirrors
Downloading the Sage mirror list
Searching fastest mirror
  177ms: http://files.sagemath.org/
  399ms: http://linorg.usp.br/sage/
   59ms: https://ftp.rediris.es/mirror/sagemath/
  523ms: https://ftp.riken.jp/sagemath/
  350ms: https://ftp.sun.ac.za/ftp/pub/mirrors/www.sagemath.org/
  264ms: https://ftp.yz.yamagata-u.ac.jp/pub/math/sage/
  174ms: https://mirror-hk.koddos.net/sagemath/
  305ms: https://mirror.aarnet.edu.au/pub/sage/
  245ms: https://mirror.csclub.uwaterloo.ca/sage/
   27ms: https://mirror.dogado.de/sage/
   12ms: https://mirror.koddos.net/sagemath/
   26ms: https://mirror.lyrahosting.com/sagemath/
   40ms: https://mirror.marwan.ma/sage/
  577ms: https://mirror.rcg.sfu.ca/mirror/sage/
  329ms: https://mirror.ufs.ac.za/sagemath/
  111ms: https://mirror.yandex.ru/mirrors/sage.math.washington.edu/
   81ms: https://mirrors.aliyun.com/sagemath/
   86ms: https://mirrors.mit.edu/sage/
  249ms: https://mirrors.nju.edu.cn/sagemath/
  685ms: https://mirrors.tuna.tsinghua.edu.cn/sagemath/
   55ms: https://mirrors.up.pt/pub/sage/
  245ms: https://mirrors.ustc.edu.cn/sagemath/
  126ms: https://mirrors.xmission.com/sage/
   40ms: https://sage.mirror.garr.it/mirrors/sage/
   15ms: https://www-ftp.lip6.fr/pub/math/sagemath/
   87ms: https://www.mirrorservice.org/sites/www.sagemath.org/
Fastest mirror: https://mirror.koddos.net/sagemath/
https://mirror.koddos.net/sagemath/spkg/upstream/configure/configure-b466c8ae02c04d287d28d2d669e1d111df3724ff.tar.gz
[......................................................................]
% grep sage_conf.py configure*
configure:ac_config_files="$ac_config_files pkgs/sage-conf/sage_conf.py pkgs/sage-conf/setup.cfg"
configure:    "pkgs/sage-conf/sage_conf.py") CONFIG_FILES="$CONFIG_FILES pkgs/sage-conf/sage_conf.py" ;;
% source .homebrew-build-env
% ./configure
[...]
zn_poly-0.9.2:                               no suitable system package; standard, will be installed as an SPKG
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating build/make/Makefile-auto
config.status: creating build/make/Makefile
config.status: creating src/bin/sage-env-config
config.status: creating src/bin/sage-src-env-config
config.status: creating build/bin/sage-build-env-config
config.status: error: cannot find input file: `pkgs/sage-conf/sage_conf.py.in'
%

mkoeppe · 2022-06-26T22:56:39Z

comment:24

From the output:

bootstrap:74: installing 'config/config.rpath'
./bootstrap: line 164: aclocal: command not found
Bootstrap failed, downloading required files instead.

These downloaded files are unsuitable - they don't fit with the changes in the ticket.

Install the bootstrapping prerequisites (https://doc.sagemath.org/html/en/reference/spkg/_bootstrap.html#spkg-bootstrap)

GMS103 · 2022-06-26T23:13:29Z

comment:25

Thanks Matthias and François.

Now I understand what happened.
I had uninstalled Homebrew because of some problems, and I thought I had reinstalled all that was needed. But I was wrong.

Once again, sorry for the noise.

kiwifb · 2022-07-01T01:29:07Z

comment:26

Volker is trying to merge the ticket and I ended finding a problem. Not sure if it will have an impact on the mergability.

$ sage-config
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.10/sage-config", line 5, in <module>
    from sage_conf.__main__ import _main
ModuleNotFoundError: No module named 'sage_conf.__main__'; 'sage_conf' is not a package

It originates from setup.cfg

[options.entry_points]
console_scripts =
    sage-config = sage_conf.__main__:_main

I tried to change the last line to sage-config = _sage_conf.__main__:_main but it leads to a new error

$ sage-config
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.10/sage-config", line 5, in <module>
    from _sage_conf.__main__ import _main
  File "/usr/lib/python3.10/site-packages/_sage_conf/__main__.py", line 3, in <module>
    from sage_conf import *
  File "/usr/lib/python3.10/site-packages/sage_conf.py", line 2, in <module>
    from _sage_conf.__main__ import _main
ImportError: cannot import name '_main' from partially initialized module '_sage_conf.__main__' (most likely due to a circular import) (/usr/lib/python3.10/site-packages/_sage_conf/__main__.py)

mkoeppe · 2022-07-01T01:53:52Z

comment:27

sage_conf is definitely a package after this ticket. Are you sure you installed it?

kiwifb · 2022-07-01T01:57:08Z

comment:28

These are the file I have installed

/usr/lib/python3.10/site-packages/__pycache__
/usr/lib/python3.10/site-packages/__pycache__/sage_conf.cpython-310.opt-1.pyc
/usr/lib/python3.10/site-packages/__pycache__/sage_conf.cpython-310.opt-2.pyc
/usr/lib/python3.10/site-packages/__pycache__/sage_conf.cpython-310.pyc
/usr/lib/python3.10/site-packages/_sage_conf
/usr/lib/python3.10/site-packages/_sage_conf/__main__.py
/usr/lib/python3.10/site-packages/_sage_conf/__pycache__
/usr/lib/python3.10/site-packages/_sage_conf/__pycache__/__main__.cpython-310.opt-1.pyc
/usr/lib/python3.10/site-packages/_sage_conf/__pycache__/__main__.cpython-310.opt-2.pyc
/usr/lib/python3.10/site-packages/_sage_conf/__pycache__/__main__.cpython-310.pyc
/usr/lib/python3.10/site-packages/_sage_conf/__pycache__/_conf.cpython-310.opt-1.pyc
/usr/lib/python3.10/site-packages/_sage_conf/__pycache__/_conf.cpython-310.opt-2.pyc
/usr/lib/python3.10/site-packages/_sage_conf/__pycache__/_conf.cpython-310.pyc
/usr/lib/python3.10/site-packages/_sage_conf/_conf.py
/usr/lib/python3.10/site-packages/sage_conf-9.7b3.dist-info
/usr/lib/python3.10/site-packages/sage_conf-9.7b3.dist-info/METADATA
/usr/lib/python3.10/site-packages/sage_conf-9.7b3.dist-info/RECORD
/usr/lib/python3.10/site-packages/sage_conf-9.7b3.dist-info/WHEEL
/usr/lib/python3.10/site-packages/sage_conf-9.7b3.dist-info/top_level.txt
/usr/lib/python3.10/site-packages/sage_conf.py

Could it be some semantics issue? sage_conf.py does not have a main section. Or may be it shouldn't be present?

mkoeppe · 2022-07-01T02:03:01Z

comment:29

Replying to @mkoeppe:

sage_conf is definitely a package after this ticket.

Sorry, I take this back

sagetrac-git · 2022-07-01T02:10:28Z

Changed commit from 9d10055 to eeddb03

sagetrac-git · 2022-07-01T02:10:28Z

Branch pushed to git repo; I updated commit sha1. New commits:

`eeddb03`	`pkgs/sage-conf/setup.cfg: Fix up console_scripts`

kiwifb · 2022-07-01T02:50:45Z

comment:33

Much better. Although I am not sure I will keep sage-config in sage-on-gentoo. We may want a more graceful behavior for variables not found in some follow up ticket.

mkoeppe · 2022-07-01T17:41:07Z

comment:34

Thanks!

vbraun · 2022-07-09T22:33:43Z

Changed branch from u/mkoeppe/refactor_sage_conf to eeddb03

mkoeppe added this to the sage-9.6 milestone Feb 5, 2022

mkoeppe added c: scripts labels Feb 5, 2022

mkoeppe added the s: needs review label Feb 5, 2022

mkoeppe modified the milestones: sage-9.6, sage-9.7 Mar 19, 2022

kiwifb added s: positive review and removed s: needs review labels Jun 21, 2022

kiwifb added s: needs work and removed s: positive review labels Jun 21, 2022

mkoeppe added s: needs review and removed s: needs work labels Jun 25, 2022

kiwifb added s: positive review and removed s: needs review labels Jun 25, 2022

mkoeppe added s: needs work and removed s: positive review labels Jul 1, 2022

mkoeppe added s: needs review and removed s: needs work labels Jul 1, 2022

kiwifb added s: positive review and removed s: needs review labels Jul 1, 2022

vbraun removed the s: positive review label Jul 9, 2022

vbraun closed this as completed in 80d2ee6 Jul 9, 2022

Refactor sage_conf #33295

Refactor sage_conf #33295

Comments

mkoeppe commented Feb 5, 2022

mkoeppe commented Feb 5, 2022

mkoeppe commented Feb 5, 2022

mkoeppe commented Feb 5, 2022

mkoeppe commented Feb 5, 2022

kiwifb commented Feb 5, 2022

mkoeppe commented Feb 5, 2022

sagetrac-git mannequin commented May 3, 2022

sagetrac-git mannequin commented May 3, 2022

dimpase commented Jun 17, 2022

sagetrac-git mannequin commented Jun 17, 2022

sagetrac-git mannequin commented Jun 17, 2022

kiwifb commented Jun 21, 2022

kiwifb commented Jun 21, 2022

kiwifb commented Jun 21, 2022

mkoeppe commented Jun 25, 2022

sagetrac-git mannequin commented Jun 25, 2022

sagetrac-git mannequin commented Jun 25, 2022

mkoeppe commented Jun 25, 2022

GMS103 commented Jun 26, 2022

GMS103 commented Jun 26, 2022

kiwifb commented Jun 26, 2022

GMS103 commented Jun 26, 2022

kiwifb commented Jun 26, 2022

GMS103 commented Jun 26, 2022

kiwifb commented Jun 26, 2022

GMS103 commented Jun 26, 2022

mkoeppe commented Jun 26, 2022

GMS103 commented Jun 26, 2022

kiwifb commented Jul 1, 2022

mkoeppe commented Jul 1, 2022

kiwifb commented Jul 1, 2022

mkoeppe commented Jul 1, 2022

sagetrac-git mannequin commented Jul 1, 2022

sagetrac-git mannequin commented Jul 1, 2022

kiwifb commented Jul 1, 2022

mkoeppe commented Jul 1, 2022

vbraun commented Jul 9, 2022