sage-download-file: Fix certificate problems with https downloads from upstream_url when sage-system-python is XCode's python3 on macOS

[mkoeppe]

#26351 added an optional upstream_url field to build/pkgs/*/checksums.ini. It streamlines the procedure for testing upgrade tickets: Developers or automatic testing facilities can pass an extra flag -o to sage-spkg to allow downloading from upstream rather than from Sage mirrors (where the updated ticket will be made available later only).

Many upstream package URLs use the https protocol - in contrast to the http protocol used when downloading from the Sage mirrors. The downloading is done via build/bin/sage-download-file, which uses the urllib module. It supports the https protocol.

However, SSL certificate problems are common on test systems. For example, if one uses XCode's python3 as the system python, then urllib does not automatically uses the standard system certificates. (This is apparently a known issue -- which is considered "wontfix" by Apple as reported here:
HandBrake/HandBrake#2216 (comment))

We add an option --no-check-certificate to sage-download-file, disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification).

Developers can set this option using the environment variable SAGE_DOWNLOAD_FILE_OPTIONS when installing packages (either by make or by using sage -i).

We note that even with SSL certificates disabled, there is still cryptographic protection because of the checksums recorded in checksums.ini.

---

Other possible workarounds considered:

• switching from using urllib directly to the requests library
• passing cafile=..., capath=... to urllib, perhaps coming from an environment variable
• using python instead of python3 on macOS as system python

As of #29090 (sage-system-python fixup) prefers /usr/bin/python3 over /usr/bin/python, leading to:

  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

(see https://github.com/mkoeppe/sage/runs/538432620)

CC: @vbraun @dimpase @kiwifb @jhpalmieri @videlec @fchapoton @kliem

Component: build

Author: Matthias Koeppe

Branch: 90ea00b

Reviewer: Jonathan Kliem

Issue created by migration from https://trac.sagemath.org/ticket/29418

[mkoeppe]

comment:2

https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification

[mkoeppe]

comment:3

https://stackoverflow.com/questions/57630314/ssl-certificate-verify-failed-error-with-python3-on-macos-10-15

[mkoeppe]

Branch: u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3

[dimpase]

comment:6

that's why I suggested not to bother with MacOS/Xcode's python(3).
Can't one get Python3 from python.org in MacOS by running a script?

---

New commits:

bd24641

build/bin/sage-download-file: Add option --no-check-certificate

[dimpase]

Commit: bd24641

[mkoeppe]

Changed commit from bd24641 to none

[mkoeppe]

Changed branch from u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3 to none

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

comment:8

Replying to @dimpase:

that's why I suggested not to bother with MacOS/Xcode's python(3).
Can't one get Python3 from python.org in MacOS by running a script?

Please... the idea is to make Sage installation easier, not harder.

[mkoeppe]

comment:9

One often runs into these certificate errors also with random Linux images when I run them on Docker. So this workaround is valuable for the automatic testing in any case.

[mkoeppe]

Branch: u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3

[mkoeppe]

Commit: e143939

[mkoeppe]

comment:11

Testing this at https://github.com/mkoeppe/sage/actions/runs/65346154

---

New commits:

bd24641	build/bin/sage-download-file: Add option --no-check-certificate
e143939	build/bin/sage-spkg: Append to SAGE_DOWNLOAD_FILE_OPTIONS instead of overwriting it

[dimpase]

comment:12

Replying to @mkoeppe:

Replying to @dimpase:

that's why I suggested not to bother with MacOS/Xcode's python(3).
Can't one get Python3 from python.org in MacOS by running a script?

Please... the idea is to make Sage installation easier, not harder.

well, it seems that using system's python would make a slighly cryptographically
broken Sagemath, one way or another.

[mkoeppe]

comment:13

Replying to @dimpase:

well, it seems that using system's python would make a slighly cryptographically
broken Sagemath, one way or another.

That's not a regression from the current status on macOS, see #29291 for example

[mkoeppe]

comment:14

And, of course, the present ticket is about sage-system-python, which is only used for bootstrapping and build. Unrelated to the use of a system python3 for venv (#27824), which has a completely separate test for what it accepts.

[dimpase]

comment:15

oy gevalt, pythons everywhere!

[mkoeppe]

comment:17

Replying to @dimpase:

oy gevalt, pythons everywhere!

This will be addressed in #45678 "switch Sage from Python to Julia"

[sagetrac-git]

Changed commit from e143939 to 90ea00b

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

cf27321	build/bin/sage-download-file: Add option --no-check-certificate
90ea00b	build/bin/sage-spkg: Append to SAGE_DOWNLOAD_FILE_OPTIONS instead of overwriting it

[mkoeppe]

comment:19

Rebased, needs review

[jhpalmieri]

comment:23

Okay, I'm really confused, or my Sage installation is broken, or both. If I use ./sage -i beautifulsoup4 with a build using the system Python on OS X, it fails, but not because of the issues here. Instead, I see this:

make[2]: *** No rule to make target `/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/SYSTEM/sage-9.1.beta9/local/var/lib/sage/installed/beautifulsoup4-none', needed by `all-sage'.  Stop.

If instead I run make beautifulsoup4, it succeeds. When I ran ./sage -i tornado it didn't seem to even try to install it, but ./sage -f tornado worked.

So: why does ./sage -i ... not work? And what is the point of this ticket: what problem does it solve?

[mkoeppe]

comment:24

sage -i was changed in #29113 -- unrelated to the present ticket

[mkoeppe]

comment:25

Could you try if the problem goes away if you run bootstrap?

[jhpalmieri]

comment:27

If I run bootstrap, then running ./sage -i benzene acts like ./sage -i tornado: it doesn't install. There is a message

running ./configure '--enable-beautifulsoup4' '--enable-biopython'  '--enable-benzene'

and then if I search my terminal window for benzene, the only other hit is

benzene-20130630:                            does not support check for system package; optional, use "./configure --enable-benzene" to install

I don't think I tested #29113 well enough when I was reviewing it.

[jhpalmieri]

comment:28

This fixes it for me:

diff --git a/src/bin/sage b/src/bin/sage
index 10acddcd96..6ccff3789c 100755
--- a/src/bin/sage
+++ b/src/bin/sage
@@ -404,7 +404,7 @@ if [ "$1" = '-i' ]; then
         #   'CC=gcc -Wall' '--enable-e_antic'
         CONFIG_CMD="./configure $(./config.status --config) $ENABLE_ARGS"
         echo >&2 "running $CONFIG_CMD"
-        bash -c "$CONFIG_CMD" && $MAKE all-build
+        bash -c "$CONFIG_CMD" && $MAKE $PACKAGES
     else
         echo "New packages may have been installed."
         echo "Re-running configure and make in case any dependent packages need updating."

(or maybe it could be $MAKE all-build $PACKAGES?)

By the way, I still don't understand the goal of this ticket.

[mkoeppe]

comment:29

Let's please take the issues with sage -i to the new ticket #29481. Thanks for catching this.

[mkoeppe]

comment:31

Replying to @jhpalmieri:

By the way, I still don't understand the goal of this ticket.

Reworked the ticket description, please take a look

[jhpalmieri]

comment:32

Replying to @mkoeppe:

Replying to @jhpalmieri:

By the way, I still don't understand the goal of this ticket.

Reworked the ticket description, please take a look

Sorry, I wasn't clear enough. What system configuration and commands lead to the problem in the description? I can't reproduce the problem, so I can't tell if the solution here works.

[kliem]

comment:33

From what I understand this is a problem that only occurs when one has python3 from Xcode. This doesn't ship certify and never will (at least Apple doesn't have any intentions). So it is really difficult to establish an ssl connection with that.

Do you have successfully tested this ticket?

Do I understand correctly that the problem occurring could be theoretically fixed by manually downloading the correct packages into the upstream folder? (Yes this is not a good approach for testing environments.)

Also this flag would never be needed for a normal user as long as the sage mirrors don't use ssl?

[mkoeppe]

comment:34

Replying to @kliem:

From what I understand this is a problem that only occurs when one has python3 from Xcode. This doesn't ship certify and never will (at least Apple doesn't have any intentions). So it is really difficult to establish an ssl connection with that.

That's a correct description for macOS. But the problem also appears on Linux distributions if one does not install ca-certificates, or those are outdated.

Do you have successfully tested this ticket?

Yes, I have been using this as part of #29417 since end of March.

Do I understand correctly that the problem occurring could be theoretically fixed by manually downloading the correct packages into the upstream folder? (Yes this is not a good approach for testing environments.)

That's correct for the macOS tests using tox local. For the Linux tests with tox docker note that /upstream is in .gitignore (therefore in .dockerignore) and is therefore not provided to the container.

Also this flag would never be needed for a normal user as long as the sage mirrors don't use ssl?

That's right, normal users would not use the upstream_url at all; it is a feature for developers only.

[kliem]

Reviewer: Jonathan Kliem

[kliem]

comment:35

LGTM.

Btw, I have been using this ticket to test my tickets, e.g. here: https://github.com/kliem/sage-test-27122/actions/runs/72102779

[mkoeppe]

comment:36

Thank you!

[vbraun]

Changed branch from u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3 to 90ea00b

[mkoeppe]

comment:38

Follow up: #30950

[mkoeppe]

Changed commit from 90ea00b to none