Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many more namespace packages #35322

Merged
merged 9 commits into from
Apr 6, 2023

Conversation

mkoeppe
Copy link
Contributor

@mkoeppe mkoeppe commented Mar 21, 2023

📚 Description

As discussed in #35100 (comment), we remove many of the empty __init__.py files, turning these packages into PEP-420 implicit namespace packages.

The following packages will keep their __init__.py files because they are intended to remain ordinary packages indefinitely:

  • sage.cpython, sage.structure – because they are shipped as a whole by sagemath-objects
  • sage.doctest, sage.repl – because they are shipped as a whole by sagemath-repl
  • sage.features – because it is shipped as a whole by sagemath-environment

This is part of:

📝 Checklist

  • I have made sure that the title is self-explanatory and the description concisely explains the PR.
  • I have linked an issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation accordingly.

⌛ Dependencies

@mkoeppe mkoeppe requested a review from tornaria March 21, 2023 03:27
@mkoeppe mkoeppe self-assigned this Mar 21, 2023
@codecov-commenter
Copy link

codecov-commenter commented Mar 21, 2023

Codecov Report

Patch coverage: 75.00% and project coverage change: +0.03 🎉

Comparison is base (c00e6c2) 88.62% compared to head (dc8b3c5) 88.65%.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop   #35322      +/-   ##
===========================================
+ Coverage    88.62%   88.65%   +0.03%     
===========================================
  Files         2148     2114      -34     
  Lines       398855   398679     -176     
===========================================
- Hits        353480   353462      -18     
+ Misses       45375    45217     -158     
Impacted Files Coverage Δ
src/sage/misc/cython.py 70.35% <ø> (ø)
src/sage/misc/package_dir.py 86.55% <74.19%> (-13.45%) ⬇️
src/sage/misc/dev_tools.py 83.54% <100.00%> (ø)

... and 62 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 21, 2023

@tornaria There is only one doctest failure, for sage.misc.dev_tools.load_submodules, which is due to the long-standing issue python/cpython#73444

I don't know if we care much about this function, but we can repair it using our function sage.misc.package_dir.is_package_or_sage_namespace_package_dir.

@tornaria
Copy link
Contributor

  1. issue with sage.symbolic.integration
sage: import sage.symbolic.integration.integral
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 1
----> 1 import sage.symbolic.integration.integral

ModuleNotFoundError: No module named 'sage.symbolic.integration'

I think you need to touch src/sage/symbolic/integration/all.py

  1. doctest failure in src/sage/misc/package_dir.py
sage -t --long --random-seed=247137144307186708220921933273202299041 src/sage/misc/package_dir.py
**********************************************************************
File "src/sage/misc/package_dir.py", line 180, in sage.misc.package_dir.is_package_or_sage_namespace_package_dir
Failed example:
    directory = os.path.join(os.path.dirname(sage.symbolic.__file__), 'ginac'); directory
Exception raised:
    Traceback (most recent call last):
      File "/home/tornaria/src/sage/sage-git/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/tornaria/src/sage/sage-git/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package_dir.is_package_or_sage_namespace_package_dir[7]>", line 1, in <module>
        directory = os.path.join(os.path.dirname(sage.symbolic.__file__), 'ginac'); directory
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<frozen posixpath>", line 152, in dirname
    TypeError: expected str, bytes or os.PathLike object, not NoneType
**********************************************************************
File "src/sage/misc/package_dir.py", line 182, in sage.misc.package_dir.is_package_or_sage_namespace_package_dir
Failed example:
    is_package_or_sage_namespace_package_dir(directory)
Expected:
    False
Got:
    True
**********************************************************************
1 item had failures:
   2 of  10 in sage.misc.package_dir.is_package_or_sage_namespace_package_dir
    [29 tests, 2 failures, 0.02 s]
----------------------------------------------------------------------

The following fixes it for me:

diff --git a/src/sage/misc/package_dir.py b/src/sage/misc/package_dir.py
index bb4e34d5c26..d4e5cb3bc7a 100644
--- a/src/sage/misc/package_dir.py
+++ b/src/sage/misc/package_dir.py
@@ -155,7 +155,7 @@ def is_package_or_sage_namespace_package_dir(path, *, distribution_filter=None):
     :mod:`sage.cpython` is an ordinary package::
 
         sage: from sage.misc.package_dir import is_package_or_sage_namespace_package_dir
-        sage: directory = os.path.dirname(sage.cpython.__file__); directory
+        sage: directory = sage.cpython.__path__[0]; directory
         '.../sage/cpython'
         sage: is_package_or_sage_namespace_package_dir(directory)
         True
@@ -163,21 +163,21 @@ def is_package_or_sage_namespace_package_dir(path, *, distribution_filter=None):
     :mod:`sage.libs.mpfr` only has an ``__init__.pxd`` file, but we consider
     it a package directory for consistency with Cython::
 
-        sage: directory = os.path.join(os.path.dirname(sage.libs.all.__file__), 'mpfr'); directory
+        sage: directory = os.path.join(sage.libs.__path__[0], 'mpfr'); directory
         '.../sage/libs/mpfr'
         sage: is_package_or_sage_namespace_package_dir(directory)
         True
 
     :mod:`sage` is designated to become an implicit namespace package::
 
-        sage: directory = os.path.dirname(sage.env.__file__); directory
+        sage: directory = sage.__path__[0]; directory
         '.../sage'
         sage: is_package_or_sage_namespace_package_dir(directory)
         True
 
     Not a package::
 
-        sage: directory = os.path.join(os.path.dirname(sage.symbolic.__file__), 'ginac'); directory
+        sage: directory = os.path.join(sage.symbolic.__path__[0], 'ginac'); directory
         '.../sage/symbolic/ginac'
         sage: is_package_or_sage_namespace_package_dir(directory)
         False
  1. doctest failure in src/sage/symbolic/pynac.pxi:
sage -t --long --random-seed=167829029651056083008002563228240301164 src/sage/symbolic/pynac.pxi
**********************************************************************
File "src/sage/symbolic/pynac.pxi", line 6, in sage.symbolic.pynac
Failed example:
    cython(  # long time; random compiler warnings  # optional - sage.misc.cython
    '''
    from sage.symbolic cimport expression
    ''')
Exception raised:
    Traceback (most recent call last):
      File "/home/tornaria/src/sage/sage-git/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/misc/cython.py", line 358, in cython
        ext, = cythonize([ext],
               ^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1112, in cythonize
        cythonize_one(*args)
      File "/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1235, in cythonize_one
        raise CompileError(None, pyx_file)
    Cython.Compiler.Errors.CompileError: /tmp/tmpefpazx19/spyx/_tmp_tmp9_e5gg0j_tmp_hmvet0wv_pyx/_tmp_tmp9_e5gg0j_tmp_hmvet0wv_pyx_0.pyx

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/tornaria/src/sage/sage-git/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/tornaria/src/sage/sage-git/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.symbolic.pynac[0]>", line 1, in <module>
        cython(  # long time; random compiler warnings  # optional - sage.misc.cython
      File "sage/misc/lazy_import.pyx", line 404, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:4483)
        return self.get_object()(*args, **kwds)
      File "/home/tornaria/src/sage/sage-git/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/misc/cython.py", line 673, in cython_compile
        return cython_import_all(tmpfile, get_globals(), **kwds)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/tornaria/src/sage/sage-git/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/misc/cython.py", line 563, in cython_import_all
        m = cython_import(filename, **kwds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/tornaria/src/sage/sage-git/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/misc/cython.py", line 534, in cython_import
        name, build_dir = cython(filename, **kwds)
                          ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/tornaria/src/sage/sage-git/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/misc/cython.py", line 374, in cython
        raise RuntimeError(cython_messages.strip())
    RuntimeError: Error compiling Cython file:
    ------------------------------------------------------------
    ...

    from sage.symbolic cimport expression
    ^
    ------------------------------------------------------------

    _tmp_tmp9_e5gg0j_tmp_hmvet0wv_pyx_0.pyx:2:0: 'sage/symbolic.pxd' not found
**********************************************************************
1 item had failures:
   1 of   2 in sage.symbolic.pynac
    [1 test, 1 failure, 0.01 s]

@tornaria
Copy link
Contributor

BTW, I noticed that the function is_package_or_sage_namespace_package_dir has an optional argument (distribution_filter) that is neither tested nor used anywhere that I could find...

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 22, 2023

    Not a package::
 
-        sage: directory = os.path.join(os.path.dirname(sage.symbolic.__file__), 'ginac'); directory
+        sage: directory = os.path.join(sage.symbolic.__path__[0], 'ginac'); directory
         '.../sage/symbolic/ginac'
         sage: is_package_or_sage_namespace_package_dir(directory)
         False

I think only this last example needs to be adjusted; the preceding examples are modules or ordinary packages.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 22, 2023

from sage.symbolic cimport expression
^
------------------------------------------------------------

_tmp_tmp9_e5gg0j_tmp_hmvet0wv_pyx_0.pyx:2:0: 'sage/symbolic.pxd' not found

OK, this is from an unpatched Cython without namespace package support.
I guess I'll wrap a with cython_namespace_package_support() around it.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 22, 2023

I guess I'll wrap a with cython_namespace_package_support() around it.

Hm, no, that is already used in sage.misc.cython.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 22, 2023

I can't reproduce this failure here, also not with stock Cython 0.29.33 installed via pip

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 22, 2023

I noticed that the function is_package_or_sage_namespace_package_dir has an optional argument (distribution_filter) that is neither tested nor used anywhere that I could find...

I think the first use of that is on one of the pending tickets from #29705. I'll check later

@tornaria
Copy link
Contributor

    Not a package::
 
-        sage: directory = os.path.join(os.path.dirname(sage.symbolic.__file__), 'ginac'); directory
+        sage: directory = os.path.join(sage.symbolic.__path__[0], 'ginac'); directory
         '.../sage/symbolic/ginac'
         sage: is_package_or_sage_namespace_package_dir(directory)
         False

I think only this last example needs to be adjusted; the preceding examples are modules or ordinary packages.

Yes, but using __path__[0] seems more robust as it will work for either ordinary or namespace packages. It's shorter, and it's really what you mean, i.e. path to module mpfr inside package sage.libs.

@tornaria
Copy link
Contributor

from sage.symbolic cimport expression
^

_tmp_tmp9_e5gg0j_tmp_hmvet0wv_pyx_0.pyx:2:0: 'sage/symbolic.pxd' not found

OK, this is from an unpatched Cython without namespace package support. I guess I'll wrap a with cython_namespace_package_support() around it.

I'll have a look and patch our Cython if it's necessary.

@tornaria
Copy link
Contributor

I can't reproduce this failure here, also not with stock Cython 0.29.33 installed via pip

Did you test with --long ? Our cython is stock 0.29.33, unpatched, compiled from pypi source using setuptools, nothing strange AFAICT. I will add the namespaces patch and I guess everything will work for us, although I guess it'd be nice to support an unpatched cython (?).

Maybe with cython_namespace_package_support() does not work as expected? It's not tested, and maybe nothing really cimported from a namespace package before this PR.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 23, 2023

I can't reproduce this failure here, also not with stock Cython 0.29.33 installed via pip

Did you test with --long ?

Oh... now I can reproduce it, thanks...

@tornaria
Copy link
Contributor

I can't reproduce this failure here, also not with stock Cython 0.29.33 installed via pip

Did you test with --long ?

Oh... now I can reproduce it, thanks...

I patched cython with https://github.com/sagemath/sage/blob/develop/build/pkgs/cython/patches/4918.patch, but I'm still getting the same failure. Either with 10.0.beta5 + 35322, or just with my system sagemath 9.8 when I remove the empty .../sage/symbolic/__init__.py. Something is not working with pep 420 support... Cython ends up complaining that .../sage/symbolic.pxd is not found. Doesn't matter if it's stock cython or patched cython, the failure is the same.

@tornaria
Copy link
Contributor

Also, note that

cython("cimport sage.symbolic.expression")

works, but

cython("cimport sage.symbolic")

doesn't (same failure as from sage.symbolic cimport expression).

Is the second one supposed to work? Both that one and the original failure work ok when .../sage/symbolic/__init__.py is present, so I'm guessing the answer is yes.

In the same style, should cimport sage work?

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 23, 2023

Is the second one supposed to work?

It would be worth checking what Cython 3 thinks about this.

@tornaria
Copy link
Contributor

Summary:

  • cimport sage.symbolic.expression needs either the patch (4918.patch) or the cython_namespace_package_support() hack.
  • from sage.symbolic.expression cimport is_Expression works fine, no need for patch or hack.
  • from sage.symbolic import expression this is broken and it's not fixed neither with the patch nor with the hack.

Here is a short term workaround:

--- a/src/sage/symbolic/pynac.pxi
+++ b/src/sage/symbolic/pynac.pxi
@@ -3,9 +3,9 @@ Declarations for pynac, a Python frontend for ginac
 
 Check that we can externally cimport this (:trac:`18825`)::
 
-    sage: cython(  # long time; random compiler warnings  # optional - sage.misc.cython
+    sage: cython(  # optional - sage.misc.cython
     ....: '''
-    ....: from sage.symbolic cimport expression
+    ....: cimport sage.symbolic.expression
     ....: ''')
 """
 

I removed the # long time and random since I don't think it's necessary, you may know better...

The long term solution is for cython to fix its pep 420 support.

Meanwhile, it would be nice to add the following tests:

---- a/src/sage/misc/cython.py
+++ b/src/sage/misc/cython.py
@@ -211,6 +211,28 @@ def cython(filename, verbose=0, compile_message=False,
         sage: cython('''
         ....: cdef size_t foo = 3/2
         ....: ''')
+
+    Check that Cython supports PEP 420 packages::
+
+        sage: cython('''
+        ....: cimport sage.misc.cachefunc
+        ....: ''')
+
+        sage: cython('''
+        ....: from sage.misc.cachefunc cimport cache_key
+        ....: ''')
+
+    In Cython 0.29.33 using `from PACKAGE cimport MODULE` is broken
+    when `PACKAGE` is a namespace package, see :trac:`35322`::
+
+        sage: cython('''
+        ....: from sage.misc cimport cachefunc
+        ....: ''')
+        Traceback (most recent call last):
+        ...
+        RuntimeError: Error compiling Cython file:
+        ...
+        ...: 'sage/misc.pxd' not found
     """
     if not filename.endswith('pyx'):
         print("Warning: file (={}) should have extension .pyx".format(filename), file=sys.stderr)

The last one is the broken one, this will warn us when it gets fixed...

@tornaria
Copy link
Contributor

Also, just cimport sage.misc is broken atm, and it should also work (the point is that namespace packages should be functionally identically to standard packages, so anything that works when an empty __init__.py file is there, should also work without it). Maybe we should also add a test for that situation and corresponding explanation (i.e. cimport PACKAGE is broken when PACKAGE is a namespace package, just like from PACKAGE cimport MODULE).

@tornaria
Copy link
Contributor

Is the second one supposed to work?

It would be worth checking what Cython 3 thinks about this.

Still broken in the same way as in 0.29.33. I'll submit an issue.

What is broken: cimporting a namespace package, and it seems sage doesn't do that (except for the doctest in pynac.pxi which can be changed as I suggested above).

What works ok: [1] is cimporting a module (or a standard package) when there is a namespace package in its path, which is what we are requiring a lot after converting all these packages to namespace packages.

[1] either with 4918.patch, with the cython_namespace_package_support() trick, or with 3.0.0b1.

Let's just move on, if you are ok with my suggestions above.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 25, 2023

Excellent, thanks very much. Yes, I agree with your fixes; I'll put them on the branch.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 25, 2023

OK, I will push these changes too

@tornaria
Copy link
Contributor

OK, I will push these changes too

Thanks, I'll clean up my worktree and test the PR as it is now, just to double check everything is ok, then approve since everything looks good now.

@tornaria
Copy link
Contributor

For the record, list of empty __init__ files after this PR:

$ find src/sage -size 0 | grep __init__ | sort
src/sage/docs/__init__.py
src/sage/groups/perm_gps/partn_ref/__init__.py
src/sage/groups/perm_gps/partn_ref2/__init__.py
src/sage/libs/arb/__init__.py
src/sage/libs/coxeter3/__init__.py
src/sage/libs/cremona/__init__.py
src/sage/libs/eclib/__init__.py
src/sage/libs/flint/__init__.py
src/sage/libs/glpk/__init__.py
src/sage/libs/gmp/__init__.py
src/sage/libs/gsl/__init__.py
src/sage/libs/lcalc/__init__.py
src/sage/libs/linbox/__init__.py
src/sage/libs/linkages/__init__.py
src/sage/libs/linkages/padics/__init__.py
src/sage/libs/linkages/padics/relaxed/__init__.py
src/sage/libs/lrcalc/__init__.py
src/sage/libs/mpmath/__init__.py
src/sage/libs/mwrank/__init__.py
src/sage/libs/pynac/__init__.py
src/sage/libs/singular/__init__.py
src/sage/libs/symmetrica/__init__.py
src/sage/repl/display/__init__.py
src/sage/repl/ipython_kernel/__init__.py
src/sage/structure/proof/__init__.py
src/sage/tests/__init__.py
src/sage/tests/books/computational-mathematics-with-sagemath/__init__.py
src/sage/tests/books/computational-mathematics-with-sagemath/sol/__init__.py
src/sage/tests/books/judson-abstract-algebra/__init__.py

What is the criteria to keep these?

@github-actions
Copy link

Documentation preview for this PR is ready! 🎉
Built with commit: dc8b3c5

@tornaria
Copy link
Contributor

Everything works ok for me on dc8b3c5.

I'm approving, although I'm still wondering about the 29 empty __init__.py files I listed above.

In addition, among the 25 nonempty __init__.* there are 5 that seem equivalent to empty:

  • src/sage/combinat/words/__init__.py -- just a docstring
  • src/sage/ext/interpreters/__init__.py -- autogenerated, just a comment
  • src/sage/libs/polybori/__init__.pxd -- just a comment ("to prevent repo corruption" ?)
  • src/sage/modules/with_basis/__init__.py -- just a docstring
  • src/sage/structure/__init__.py-- resolve a cyclic import (?)

@tornaria
Copy link
Contributor

About my original concern of package conflicts between system installed sagemath vs. user installed one, I propose the following approach:

  • For system installed sagemath, we ask distro maintainers to remove every empty __init__.py file, so they will never show up in the system site-packages, i.e. they are namespace packages in the system tree.
  • From now on when some package in sagemath needs to be switched from "standard" to "namespace" proceed as follows:
    • if the package is standard because of an empty __init__.* file, just remove the file
    • if the package is standard because of a non-empty __init__.* file, require two releases of sagemath: in the first release, everything in the __init__.* file is rearranged as necessary, but an empty __init__.py file is kept; in the second release, the empty __init__.py file can be removed.

This ensures that we avoid conflicts with system packages. Anything having an empty __init__.py file is a standard package in a user installation, but a namespace package in system installation, so it can be safely switched to fully namespace package. Anything having a non-trivial __init__.* file is a standard package in system installation so it can't be switched to namespace package in development versions right away (has to wait for a new release so distros have time to adjust)

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 25, 2023

Thanks! Sorry for the slowness in responding to your comments.

For the record, list of empty __init__ files after this PR:

$ find src/sage -size 0 | grep __init__ | sort
src/sage/docs/__init__.py

sage.docs only has deprecated stuff in it and will disappear soon.

src/sage/groups/perm_gps/partn_ref/__init__.py
src/sage/groups/perm_gps/partn_ref2/__init__.py

I kept these because they looked "self-contained" to me, but it's fine with me to remove them.

src/sage/libs/arb/__init__.py
src/sage/libs/coxeter3/__init__.py
[...]
src/sage/libs/singular/__init__.py
src/sage/libs/symmetrica/__init__.py

Quite likely, each of these can remain an ordinary package; we can look into this in more detail in a follow up.

src/sage/repl/display/__init__.py
src/sage/repl/ipython_kernel/__init__.py
src/sage/structure/proof/__init__.py

All of sage.repl and all of sage.structure is intended to remain monolithic (see PR description above); and namespace packages within ordinary packages make no sense

src/sage/tests/__init__.py
src/sage/tests/books/computational-mathematics-with-sagemath/__init__.py
src/sage/tests/books/computational-mathematics-with-sagemath/sol/__init__.py
src/sage/tests/books/judson-abstract-algebra/__init__.py

Kept these because I don't have a plan for what to do with sage.tests yet. Likely to remain monolithic.

In addition, among the 25 nonempty __init__.* there are 5 that seem equivalent to empty:

  • src/sage/combinat/words/__init__.py -- just a docstring

OK, we can take care of the docstring using sage.misc.namespace_package.install_doc

  • src/sage/ext/interpreters/__init__.py -- autogenerated, just a comment

Follow-up PR

  • src/sage/libs/polybori/__init__.pxd -- just a comment ("to prevent repo corruption" ?)

Monolithic

  • src/sage/modules/with_basis/__init__.py -- just a docstring
  • src/sage/structure/__init__.py-- resolve a cyclic import (?)

Monolithic by design

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 25, 2023

About my original concern of package conflicts between system installed sagemath vs. user installed one, I propose the following approach: [...]

Sounds good to me. PR to add this to src/doc/en/developer/packaging_sage_library.rst?

Copy link
Contributor

@tornaria tornaria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm removing my approval for this until all the troublesome interaction between system sagemath and development sagemath is sorted out.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Apr 1, 2023

@tornaria, the idea of such a configuration -- old incompatible Sage installed in system package, and hoping that the development Sage shadows it properly -- is just fundamentally flawed and cannot be supported. It's not just namespace packages vs old ordinary packages. Also when a Python module in the new version replaces a Cython module of the same name (as happened recently with sage.geometry.lattice_points), the old Cython module would continue to win.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Apr 1, 2023

As I wrote to your privately, if you need this for some reason (likely because your platform or workflow is not compatible with / aware of venvs), just putting an empty file src/sage/__init__.py will make sage an ordinary package, which will shadow everything that comes after it in the search path.

@tornaria
Copy link
Contributor

tornaria commented Apr 1, 2023

As I wrote to your privately, if you need this for some reason (likely because your platform or workflow is not compatible with / aware of venvs), just putting an empty file src/sage/__init__.py will make sage an ordinary package, which will shadow everything that comes after it in the search path.

I will test this and report back.

The statement that there is no way to support a (quite up to date) system package of sagemath simultaneously with a development version of sagemath where I don't have to build infinite stuff other than the sagelib itself, it's very discouraging.

I'm willing to change my workflow, as long as it doesn't mean using a local "venv" in which I have to replicate all python packages that I already have installed in my system. It seems your idea will solve that.

Seriously, my workflow is very, very, lean, and it's a world of difference compared to anything else I knew before, that I'd rather stop using sage than go back to ./configure ; make (not to mention the download size). I think it takes less time to build sagelib (from a clean git checkout) than the time it takes just to run ./configure.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Apr 1, 2023 via email

@tornaria
Copy link
Contributor

tornaria commented Apr 4, 2023

The solution of creating an empty sage/__init__.py file in the sagelib I'm testing works perfectly to isolate it from an "old" sage installed in the system site-packages.

I've tested both this PR and #35366 in this way, with sagemath 9.8 installed from system, without having to adjust anything in the system package.

Thus, I'm giving back my positive review / approval for this PR, which I had already looked at. Thanks for your patience.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Apr 4, 2023

Glad that this is working for you! Thanks for the review.

@vbraun vbraun merged commit 11cad42 into sagemath:develop Apr 6, 2023
vbraun pushed a commit that referenced this pull request Apr 6, 2023
    
<!-- Please provide a concise, informative and self-explanatory title.
-->
<!-- Don't put issue numbers in the title. Put it in the Description
below. -->
<!-- For example, instead of "Fixes #12345", use "Add a new method to
multiply two integers" -->

### 📚 Description

<!-- Describe your changes here in detail. -->
Follow-up from #35322.
<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes #12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. It should be `[x]` not `[x
]`. -->

- [x] The title is concise, informative, and self-explanatory.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- #12345: short description why this is a dependency
- #34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: #35366
Reported by: Matthias Köppe
Reviewer(s): Gonzalo Tornaría
vbraun pushed a commit that referenced this pull request Apr 6, 2023
    
<!-- Please provide a concise, informative and self-explanatory title.
-->
<!-- Don't put issue numbers in the title. Put it in the Description
below. -->
<!-- For example, instead of "Fixes #12345", use "Add a new method to
multiply two integers" -->

### 📚 Description

This is a follow-up on:
- #35110

As preparation for #35322, which is changing more packages to implicit
namespace packages, we remove `.all` imports from these packages
throughout the Sage library.

This is part of:
- #29705

<!-- Describe your changes here in detail. -->
<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes #12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. It should be `[x]` not `[x
]`. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- #12345: short description why this is a dependency
- #34567: ...
-->
- Depends on #35418
- Depends on #35358
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: #35372
Reported by: Matthias Köppe
Reviewer(s): Gonzalo Tornaría
@mkoeppe mkoeppe added this to the sage-10.0 milestone Apr 7, 2023
@mkoeppe mkoeppe deleted the many_more_namespace_packages branch April 7, 2023 17:58
@tobiasdiez tobiasdiez mentioned this pull request Nov 23, 2023
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants