Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sage.repl: Replace use of SAGE_EXTCODE by importlib.resources #31306

Closed
mkoeppe opened this issue Jan 30, 2021 · 31 comments
Closed

sage.repl: Replace use of SAGE_EXTCODE by importlib.resources #31306

mkoeppe opened this issue Jan 30, 2021 · 31 comments

Comments

@mkoeppe
Copy link
Member

mkoeppe commented Jan 30, 2021

We eliminate direct reading of files from the package directories of sagelib by using importlib.resources (available since python 3.7)

  • git grep 'SAGE_EXTCODE' src/sage
  • git grep '__file__' src/sage

This will help make sagelib zip_safe (https://setuptools.readthedocs.io/en/latest/userguide/miscellaneous.html#setting-the-zip-safe-flag)

In this ticket, we take care of sage.repl, moving data needed by its doctests from SAGE_EXTCODE (= src/sage/ext_data) into the package directory. (Thus, we avoid a dependency on support for resources in namespace packages brought by Python 3.10(?) or the backport package importlib-resources, see python/importlib_resources#196)

Follow-up tickets will deal with other parts of the library. In the end, the directory src/sage/ext_data will be eliminated.

References:

CC: @tobiasdiez @fchapoton @kiwifb @dimpase @soehms @tscrim

Component: refactoring

Author: Matthias Koeppe

Branch/Commit: d925cd2

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/31306

@mkoeppe mkoeppe added this to the sage-9.4 milestone Jan 30, 2021
@mkoeppe mkoeppe modified the milestones: sage-9.4, sage-9.5 Jul 19, 2021
@mkoeppe

This comment has been minimized.

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 6, 2021

Changed dependencies from #30551 to #30551, #31386

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 6, 2021

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 6, 2021

Commit: b2cb173

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 6, 2021

Last 10 new commits:

77c957fsrc/doc/en/developer/packaging_sage_library.rst: Put hierarchy section one level higher
9375ef8pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels
c6fc111Prettier diagram
7d6a44cUse :mod: as markup for packages/modules
f83d424Improve ABC example
817a8d0src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup
b555735src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata
81e9c9aMerge #32899
af47b5fMerge #31386
b2cb173src/sage/repl/rich_output/: Move example files here from src/sage/ext_data, use importlib.resources

@mkoeppe mkoeppe changed the title Replace use of SAGE_EXTCODE by importlib.resources sage.repl: Replace use of SAGE_EXTCODE by importlib.resources Dec 6, 2021
@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 6, 2021

Branch pushed to git repo; I updated commit sha1. New commits:

0f63ccesrc/sage/repl/display/formatter.py: Replace use of SAGE_EXTCODE in doctest by importlib.resources

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 6, 2021

Changed commit from b2cb173 to 0f63cce

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 6, 2021

Author: Matthias Koeppe

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 6, 2021

Changed dependencies from #30551, #31386 to #31386

@mkoeppe

This comment has been minimized.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 12, 2021

Branch pushed to git repo; I updated commit sha1. New commits:

09787afMerge tag '9.5.beta8' into t/31306/replace_use_of_sage_extcode_by_importlib_resources

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 12, 2021

Changed commit from 0f63cce to 09787af

@mkoeppe

This comment has been minimized.

@mkoeppe

This comment has been minimized.

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 17, 2021

comment:13

The testsuite failures are unrelated

@dimpase
Copy link
Member

dimpase commented Dec 18, 2021

comment:14

lgtm

@dimpase
Copy link
Member

dimpase commented Dec 18, 2021

Reviewer: Dima Pasechnik

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 18, 2021

comment:15

Thanks!

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 23, 2021

Changed commit from 09787af to 9873958

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 23, 2021

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

9873958Merge tag '9.5.beta9' into t/31306/replace_use_of_sage_extcode_by_importlib_resources

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 23, 2021

comment:17

trivial merge

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jan 23, 2022

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

d925cd2Merge tag '9.5.rc3' into t/31306/replace_use_of_sage_extcode_by_importlib_resources

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jan 23, 2022

Changed commit from 9873958 to d925cd2

@mkoeppe
Copy link
Member Author

mkoeppe commented Jan 23, 2022

Changed dependencies from #31386 to none

@kiwifb
Copy link
Member

kiwifb commented Jan 31, 2022

comment:21

OK, Volker has started merging stuff for 9.6 including this. And I get

$ sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py
too many failed tests, not using stored timings
Running doctests with ID 2022-01-31-20-37-00-111581fe.
Using --optional=pip,sage
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,graphviz,imagemagick,jupymake,kenzo,latte_int,lrslib,mcqd,meataxe,pandoc,pdf2svg,plantri,pynormaliz,python_igraph,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.plot,sage.rings.number_field,sage.rings.padics,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,sphinx,tdlib
Doctesting 1 file.
sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 248, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputSceneWavefront.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[15]>", line 1, in <module>
        backend.validate(dm.types.OutputSceneWavefront.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py", line 354, in example
        scene_obj = OutputBuffer.from_file(filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example_wavefront_scene.obj'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 249, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputSceneCanvas3d.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[16]>", line 1, in <module>
        backend.validate(dm.types.OutputSceneCanvas3d.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py", line 161, in example
        return cls(OutputBuffer.from_file(filename))
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.canvas3d'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 250, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoOgg.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[17]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoOgg.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.ogv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 251, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoWebM.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[18]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoWebM.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.webm'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 252, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoMp4.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[19]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoMp4.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mp4'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 253, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoFlash.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[20]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoFlash.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.flv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 254, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoMatroska.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[21]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoMatroska.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mkv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 255, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoAvi.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[22]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoAvi.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.avi'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 256, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoWmv.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[23]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoWmv.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.wmv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 257, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoQuicktime.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[24]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoQuicktime.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mov'
**********************************************************************

And it comes down to that block https://github.com/sagemath/sage-prod/blob/develop/src/sage/repl/rich_output/buffer.py#L139 - chmod is not done if the file is in the SAGE_EXTCODE folder but executed otherwise. Of course all those files have been moved out of SAGE_EXTCODE.

@kiwifb
Copy link
Member

kiwifb commented Jan 31, 2022

comment:22

I also have

sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py  # 25 doctests failed
sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py  # 14 doctests failed

because of the same thing. I guess it doesn't fail on user space build because you can safely chmod anything.

@kiwifb
Copy link
Member

kiwifb commented Jan 31, 2022

comment:23

Follow up at #33256.

@dimpase
Copy link
Member

dimpase commented Jan 31, 2022

comment:24

Wow, doctesting of /usr/lib/python3.10/site-packages/sage/...

@kiwifb
Copy link
Member

kiwifb commented Jan 31, 2022

comment:25

Replying to @dimpase:

Wow, doctesting of /usr/lib/python3.10/site-packages/sage/...

Is it the python3.10 or the /usr/lib/ that makes you go "wow"? I can do the python3.9 flavor too if you want. I have been doctesting system installed sage and reporting from those tests for the last 10 years so I may be a bit blazé about it.

@dimpase
Copy link
Member

dimpase commented Jan 31, 2022

comment:26

it's about /usr/lib

@vbraun
Copy link
Member

vbraun commented Jan 31, 2022

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants