Recent updates to 3.9 and 3.7 images led to issue with SSL module during pip deps installation

[CarpathianUA]

WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
[1033][2023-01-19 09:56:22] Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

3.9.15 and 3.7.15 images aren't affected

[POD666]

To be more specific, the issue seems to be related to our custom kaniko image. When building with docker build .. no issue was detected.

The issue is definitely related to some recent changes in python:3.9 and python:3.7 images, until yesterday everything worked as expected. Alpine images work fine even after the recent changes.

[tianon]

Is there a particular reason you're building the image yourself instead of using the pre-built images? (the pre-built images work fine, right?)

[CarpathianUA]

@tianon We use pre-built images like python:3.9 as a base layer for our custom imagse

[tianon]

Hmm, that's weird -- all that's changed recently is #785, which shouldn't affect TLS, so I wonder what Kaniko is doing that's causing that to misbehave? 🤔 (unfortunately I don't actually know very much about Kaniko 🙈)

[POD666]

I have no deep understanding on how kaniko works but I will try to explain with my words:

• kaniko is based on scratch image by default, so it allows to build any images with any FROM statement.
• after we created custom kaniko image, we had issues building images that are not on the same "base" as our custom kaniko image
• I guess some path changes -rpath='$\$ORIGIN/../lib' lead to some conflicts between our custom kaniko image and the new python:3.9 image, but not sure why python:3.9-alpine works fine
• side effect looks like a missing library for SSL. Maybe other libs are missing too.

For anyone else who encounter such issue with custom kaniko - you could use previous python:3.9.15 image.

[tianon]

Do you have a small reproducer? @yosifkit is playing with kaniko now and pip install xxx is working fine for him 😅

[POD666]

Yes, here is a script to reproduce:

echo "
FROM gcr.io/kaniko-project/executor:v1.8.1-debug as kaniko
FROM docker:20.10.6
# Add kaniko binary
COPY --from=kaniko /kaniko/executor /kaniko/
# we install a lot of additional tools&packages but the issue is reproduced without them
" > Dockerfile.custom-kaniko

docker build -t custom-kaniko - < Dockerfile.custom-kaniko

# this works with python:3.9.15
docker run --rm -it custom-kaniko sh -c 'echo "FROM python:3.9.15" > Dockerfile.test-image; echo "RUN pip install -U pip" >> Dockerfile.test-image; /kaniko/executor --dockerfile Dockerfile.test-image --no-push;'

# this fails with python:3.9
# Could not fetch URL https://pypi.org/simple/pip/:
#    There was a problem confirming the ssl certificate:
#    HTTPSConnectionPool(host='pypi.org', port=443):
#       Max retries exceeded with url: /simple/pip/ (
#            Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")
#       ) - skipping
docker run --rm -it custom-kaniko sh -c 'echo "FROM python:3.9" > Dockerfile.test-image; echo "RUN pip install -U pip" >> Dockerfile.test-image; /kaniko/executor --dockerfile Dockerfile.test-image --no-push;'

[albertok]

Started having the same issue yesterday as well.

WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")'

Can build with Docker builder fine but pip and python complain about SSL modules under Kaniko.

Added the following to the Dockerfile to try and get more info on the state of ssl module.
RUN python -c "import ssl; print(ssl.get_default_verify_paths())"

With the following output when running in Kaniko.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/ssl.py", line 98, in <module>
    import _ssl             # if we can't import it, let the error propagate
ImportError: /usr/local/lib/python3.7/lib-dynload/_ssl.cpython-37m-x86_64-linux-gnu.so: undefined symbol: COMP_get_type, version OPENSSL_1_1_0
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

Issue seems to coincide with 2592ad8

[yosifkit]

I am unable to reproduce with regular kaniko.

$ cat Dockerfile
FROM python:3.9

RUN python -c "import ssl; print(ssl.get_default_verify_paths())"
RUN pip install -U pip

$ docker run -it --rm -v "$PWD":/workspace:ro gcr.io/kaniko-project/executor --no-push --context dir:///workspace/
Unable to find image 'gcr.io/kaniko-project/executor:latest' locally
latest: Pulling from kaniko-project/executor
6be3c5f076fd: Pull complete 
b2f199af8764: Pull complete 
1089af1ea40d: Pull complete 
765be2d750b6: Pull complete 
0a4cc742be6e: Pull complete 
eda2627b45bb: Pull complete 
e5af998bc988: Pull complete 
642c4fd59cfc: Pull complete 
2304e0ce77a3: Pull complete 
Digest: sha256:23ae6ccaba2b0f599966dbc5ecf38aa4404f4cd799add224167eaf285696551a
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
INFO[0000] Retrieving image manifest python:3.9         
INFO[0000] Retrieving image python:3.9 from registry index.docker.io 
INFO[0000] Built cross stage deps: map[]                
INFO[0000] Retrieving image manifest python:3.9         
INFO[0000] Returning cached image manifest              
INFO[0000] Executing 0 build triggers                   
INFO[0000] Building stage 'python:3.9' [idx: '0', base-idx: '-1'] 
INFO[0000] Unpacking rootfs as cmd RUN python -c "import ssl; print(ssl.get_default_verify_paths())" requires it. 
INFO[0011] RUN python -c "import ssl; print(ssl.get_default_verify_paths())" 
INFO[0011] Initializing snapshotter ...                 
INFO[0011] Taking snapshot of full filesystem...        
INFO[0013] Cmd: /bin/sh                                 
INFO[0013] Args: [-c python -c "import ssl; print(ssl.get_default_verify_paths())"] 
INFO[0013] Running: [/bin/sh -c python -c "import ssl; print(ssl.get_default_verify_paths())"] 
DefaultVerifyPaths(cafile=None, capath='/usr/lib/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs')
INFO[0013] Taking snapshot of full filesystem...        
INFO[0014] RUN pip install -U pip                       
INFO[0014] Cmd: /bin/sh                                 
INFO[0014] Args: [-c pip install -U pip]                
INFO[0014] Running: [/bin/sh -c pip install -U pip]     
Requirement already satisfied: pip in /usr/local/lib/python3.9/site-packages (22.0.4)
Collecting pip
  Downloading pip-22.3.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 33.0 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
      Successfully uninstalled pip-22.0.4
Successfully installed pip-22.3.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
INFO[0017] Taking snapshot of full filesystem...        
INFO[0018] Skipping push to container registry due to --no-push flag 

COPY --from=kaniko /kaniko/executor /kaniko/? This is not a supported way to use kaniko and I am surprised that it hasn't caused issues before now. Kaniko works by extracting the filesystem of the target image (FROM) directly into its container root (/), so if there are other files/tools there before building, then they'd likely clash with each other. Especially with docker:20.* being Alpine Linux based and python:3.9 being Debian based and extracted on top of it.

• Running kaniko in any Docker image other than the official kaniko image is not supported (ie YMMV).
  • This includes copying the kaniko executables from the official image into another image.

- https://github.com/GoogleContainerTools/kaniko/tree/fe2413e6e3c8caf943d50cf1d233a561943df1d6#known-issues

I'd love to know why rpath would cause this to break (and I think we may have a bug somewhere), but I'm not certain this is a valid build environment since it has multiple container OS's extracted on top of one another. Even though kaniko seems smart enough not to include any files from docker:20.* in the resulting image it creates, the RUN lines are still running in the context of the franken-OS.

[POD666]

This is not a supported way to use kaniko

You are right, we know it. We consider it as a performance trick (build jobs don't require dind, ~40-60s saved) and we are ready to switch this off on demand.

Anyway, we fixed our CI by pinning to python:3.9.15 before reporting the issue.
We decided that it might occur for someone else, so we shared our findings 🙂