Build ray image with ubi base

[psschwei]

Summary

Build the ray image using a UBI base

Details and comments

This currently only tests x86, as I have no way to test an ARM image. Per Aki, the UBI base images work on Apple Silicon.

I've also left the old dockerfile (using the community Ray image as a base), in case that's helpful.

This also pins numpy to v1.x, as v2 causes a number of issues

[akihikokuroda]

Where are the code for the ray head and ray worker nodes installed?

[psschwei]

Where are the code for the ray head and ray worker nodes installed?

The Ray libraries are installed as part of the client install. I'll make that clearer by adding an explicit call to install the requirements.txt file.

[akihikokuroda]

The same UBI base image has been used for the gateway. It's been working on my mac.

[psschwei]

@akihikokuroda when you have a chance (not urgent), can you build this image and run the test suite? would be good to verify it works on arm 🙏

[akihikokuroda]

I got this build error.

[+] Building 10.5s (6/12)                                                                                                                                                                                   
 => [internal] load build definition from Dockerfile-ray-node                                                                                                                                          0.0s
 => => transferring dockerfile: 1.12kB                                                                                                                                                                 0.0s
 => [internal] load metadata for registry.access.redhat.com/ubi9-minimal:9.4-1134                                                                                                                      2.4s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 480B                                                                                                                                                                      0.0s
 => CACHED [ 1/10] FROM registry.access.redhat.com/ubi9-minimal:9.4-1134@sha256:a7d837b00520a32502ada85ae339e33510cdfdbc8d2ddf460cc838e12ec5fa5a                                                       0.0s
 => => resolve registry.access.redhat.com/ubi9-minimal:9.4-1134@sha256:a7d837b00520a32502ada85ae339e33510cdfdbc8d2ddf460cc838e12ec5fa5a                                                                0.0s
 => [internal] load build context                                                                                                                                                                      1.9s
 => => transferring context: 321.09kB                                                                                                                                                                  1.8s
 => ERROR [ 2/10] RUN microdnf install -y python3.11-3.11.7-1.el9_4.1 python3.11-pip-22.3.1-5.el9 python3.11-devel-3.11.7-1.el9_4.1 vim-enhanced-2:8.2.2637-20.el9_1 wget &&    microdnf clean all     7.9s
------                                                                                                                                                                                                      
 > [ 2/10] RUN microdnf install -y python3.11-3.11.7-1.el9_4.1 python3.11-pip-22.3.1-5.el9 python3.11-devel-3.11.7-1.el9_4.1 vim-enhanced-2:8.2.2637-20.el9_1 wget &&    microdnf clean all:                
0.273                                                                                                                                                                                                       
0.273 (microdnf:7): librhsm-WARNING **: 17:31:32.996: Found 0 entitlement certificates                                                                                                                      
0.278                                                                                                                                                                                                       
0.278 (microdnf:7): librhsm-WARNING **: 17:31:33.002: Found 0 entitlement certificates
0.374 Downloading metadata...
2.715 Downloading metadata...
5.596 Downloading metadata...
7.893 error: No package matches 'python3.11-3.11.7-1.el9_4.1'
------
Dockerfile-ray-node:4
--------------------
   3 |     FROM registry.access.redhat.com/ubi9-minimal:9.4-1134
   4 | >>> RUN microdnf install -y python3.11-3.11.7-1.el9_4.1 python3.11-pip-22.3.1-5.el9 python3.11-devel-3.11.7-1.el9_4.1 vim-enhanced-2:8.2.2637-20.el9_1 wget &&\
   5 | >>>     microdnf clean all
   6 |     RUN ln -s /usr/bin/python3.11 /usr/local/bin/python3 && \
--------------------
error: failed to solve: process "/bin/sh -c microdnf install -y python3.11-3.11.7-1.el9_4.1 python3.11-pip-22.3.1-5.el9 python3.11-devel-3.11.7-1.el9_4.1 vim-enhanced-2:8.2.2637-20.el9_1 wget &&    microdnf clean all" did not complete successfully: exit code: 1
FATA[0010] no image was built                           
Error: exit status 1
make: *** [build-ray-node] Error 1

[akihikokuroda]

I'm not sure why it works for the gateway but not for the ray-node.

[psschwei]

looks like arm updated to el9_4.3 ?

(full package list: here's the package list for aarch64: https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi9/9/aarch64/appstream/os/Packages/p/)

[psschwei]

@akihikokuroda do we need to specify the version numbers for the packages we're installing? or could we get by with something like microdnf install python3.11 python3.11-pip ... instead?

[akihikokuroda]

@psschwei Yes, it builds.

[akihikokuroda]

LGTM. Thanks!

[akihikokuroda]

Static scan probably ask us to add the version :-)

Last metadata expiration check: 0:02:23 ago on Tue Jul 23 19:50:02 2024.
Installed Packages
Name         : python3.11
Version      : 3.11.7
Release      : 1.el9_4.3
Architecture : aarch64
Size         : 83 k
Source       : python3.11-3.11.7-1.el9_4.3.src.rpm
Repository   : @System
From repo    : ubi-9-appstream-rpms
Summary      : Version 3.11 of the Python interpreter
URL          : https://www.python.org/
License      : Python
Description  : Python 3.11 is an accessible, high-level, dynamically typed, interpreted
             : programming language, designed with an emphasis on code readability.
             : It includes an extensive standard library, and has a vast ecosystem of
             : third-party libraries.
             : 
             : The python3.11 package provides the "python3.11" executable: the reference
             : interpreter for the Python language, version 3.
             : The majority of its standard library is provided in the python3.11-libs package,
             : which should be installed automatically along with python3.11.
             : The remaining parts of the Python standard library are broken out into the
             : python3.11-tkinter and python3.11-test packages, which may need to be installed
             : separately.
             : 
             : Documentation for Python is provided in the python3.11-docs package.
             : 
             : Packages containing additional libraries for Python are generally named with
             : the "python3.11-" prefix.

sh-5.1# dnf info python3.11-pip
Last metadata expiration check: 0:02:55 ago on Tue Jul 23 19:50:02 2024.
Installed Packages
Name         : python3.11-pip
Version      : 22.3.1
Release      : 5.el9
Architecture : noarch
Size         : 14 M
Source       : python3.11-pip-22.3.1-5.el9.src.rpm
Repository   : @System
From repo    : ubi-9-appstream-rpms
Summary      : A tool for installing and managing Python packages
URL          : https://pip.pypa.io/
License      : MIT and Python and ASL 2.0 and BSD and ISC and LGPLv2 and MPLv2.0 and (ASL 2.0 or BSD)
Description  : pip is a package management system used to install and manage software packages
             : written in Python. Many packages can be found in the Python Package Index
             : (PyPI). pip is a recursive acronym that can stand for either "Pip Installs
             : Packages" or "Pip Installs Python".

sh-5.1# dnf info python3.11-devel
Last metadata expiration check: 0:03:15 ago on Tue Jul 23 19:50:02 2024.
Installed Packages
Name         : python3.11-devel
Version      : 3.11.7
Release      : 1.el9_4.3
Architecture : aarch64
Size         : 929 k
Source       : python3.11-3.11.7-1.el9_4.3.src.rpm
Repository   : @System
From repo    : ubi-9-appstream-rpms
Summary      : Libraries and header files needed for Python development
URL          : https://www.python.org/
License      : Python
Description  : This package contains the header files and configuration needed to compile
             : Python extension modules (typically written in C or C++), to embed Python
             : into other programs, and to make binary distributions for Python libraries.
             : 
             : It also contains the necessary macros to build RPM packages with Python modules
             : and 2to3 tool, an automatic source converter from Python 2.X.

sh-5.1# dnf info vim-enhanced    
Last metadata expiration check: 0:03:42 ago on Tue Jul 23 19:50:02 2024.
Installed Packages
Name         : vim-enhanced
Epoch        : 2
Version      : 8.2.2637
Release      : 20.el9_1
Architecture : aarch64
Size         : 3.8 M
Source       : vim-8.2.2637-20.el9_1.src.rpm
Repository   : @System
From repo    : ubi-9-appstream-rpms
Summary      : A version of the VIM editor which includes recent enhancements
URL          : http://www.vim.org/
License      : Vim and MIT
Description  : VIM (VIsual editor iMproved) is an updated and improved version of the
             : vi editor.  Vi was the first real screen-based editor for UNIX, and is
             : still very popular.  VIM improves on vi by adding new features:
             : multiple windows, multi-level undo, block highlighting and more.  The
             : vim-enhanced package contains a version of VIM with extra, recently
             : introduced features like Python and Perl interpreters.
             : 
             : Install the vim-enhanced package if you'd like to use a version of the
             : VIM editor which includes recently added enhancements like
             : interpreters for the Python and Perl scripting languages.  You'll also
             : need to install the vim-common package.

sh-5.1# dnf info wget        
Last metadata expiration check: 0:03:52 ago on Tue Jul 23 19:50:02 2024.
Installed Packages
Name         : wget
Version      : 1.21.1
Release      : 7.el9
Architecture : aarch64
Size         : 3.1 M
Source       : wget-1.21.1-7.el9.src.rpm
Repository   : @System
From repo    : ubi-9-appstream-rpms
Summary      : A utility for retrieving files using the HTTP or FTP protocols
URL          : http://www.gnu.org/software/wget/
License      : GPLv3+
Description  : GNU Wget is a file retrieval utility which can use either the HTTP or
             : FTP protocols. Wget features include the ability to work in the
             : background while you are logged out, recursive retrieval of
             : directories, file name wildcard matching, remote file timestamp
             : storage and comparison, use of Rest with FTP servers and Range with
             : HTTP servers to retrieve files over slow or unstable connections,
             : support for Proxy servers, and configurability.
             ```

[psschwei]

ok, if we have to add the version then we'll need to separate out the x86 and aarch64 builds, as there are different package versions for each architecture.

do you have a link to an example of the static scan requiring the version to be hard-coded?

[akihikokuroda]

It is (was?) the docker file linter that runs in Travis CI. This one fixed the package installed by the apt. af8d62a

[Tansito]

Static scan probably ask us to add the version :-)

I don't know the static scan specifically but security is helping us to pass all the scans for the providers without a version on it. So maybe not needed right now to try to fix the vulnerabilities that we find.

I would merge this, push the image in our registry and ask them to scan it.

[Tansito]

I could build it and run it too!

Apart from my previous comment I see everything fine 🎉

[psschwei]

I had to drop the versions from the gateway image too, but we can just send both images to security

I would merge this, push the image in our registry and ask them to scan it.