#220 SLC UX Gen 2

.github/workflows/tests-ordinary-integration.yml

@@ -16,7 +16,7 @@ jobs:
         file:
           - test_itde_manager.py
           - test_sagemaker_extension_wrapper.py
-          - test_slct_manager.py
+          - itest_slc.py
           - test_transformers_extension_wrapper.py
           - itde_mgr_in_container/test_itde_connect.py
           - itde_mgr_in_container/test_itde_external.py
@@ -36,15 +36,14 @@ jobs:
         with:
           tool-cache: true
           large-packages: false
+          haskell: true
 
       - name: Free disk space by removing large directories
         run: |
           sudo rm -rf /usr/local/graalvm/
-          sudo rm -rf /usr/local/.ghcup/
           sudo rm -rf /usr/local/share/powershell
           sudo rm -rf /usr/local/share/chromium
           sudo rm -rf /usr/local/lib/node_modules
-          sudo rm -rf /opt/ghc
 
       - name: Setup Python & Poetry Environment
         uses: exasol/python-toolbox/.github/actions/python-environment@1.7.4

doc/changes/unreleased.md

@@ -1,9 +1,32 @@
 # Unreleased
 
+## Summary
+
+This release of the Notebook Connector evolves the prior `SlctManager` interface to
+* Support building SLCs with GPU support, requiring a special flavor, e.g. [template-Exasol-8-python-3.10-cuda-conda](https://github.com/exasol/script-languages/tree/master/flavors/template-Exasol-8-python-3.10-cuda-conda)
+* Incl. isolating various SLC sessions regarding flavors, Git clones and working directories.
+
+The release comes with breaking changes:
+* Class `SlctManager` has been renamed to `ScriptLanguageContainer`.
+* Some public methods or and attributes have been
+  * renamed
+    * `flavor_name` to `flavor`
+    * `flavor_dir` to `flavor_path`
+    * `upload()` to `deploy()`
+    * `slc_docker_images()` to `docker_image_tags()`
+    * `clean_all_images()` to `clean_docker_images()`
+  * removed
+    * `check_slc_repo_complete()`
+    * `clone_slc_repo()`
+* The handling of Secure Configuration Storage has been changed to support using multiple SLC flavors.
+
+See the NC [User Guide](../user_guide/user-guide.md) for details.
+
 ## Features
 
 * #213: Added Support to specify the SLC flavor via a session parameter
 * #205: Added GPU support to ITDE manager
+* #220: Replaced implementation on using SLCs by Gen.2
 
 ## Documentation
 
@@ -16,4 +39,3 @@
 * #208: Widened version constraints for:
    * exasol-saas-api from ">=0.9.0,<1.0.0" to ">=0.9.0,<3"
    * exasol-bucketfs from "^1.0.0" to ">=1,<3"
-

doc/developer_guide/developer-guide.md

@@ -44,3 +44,9 @@ Check if the referred version of SLCR is also compatible with the version of SLC
 1. Go to [SLCR releases](https://github.com/exasol/script-languages-release/releases)
 2. Search for the SLCR version referenced in NC's file `exasol/nb_connector/slct_manager.py`
 3. Check if in the SLCR release, file  `pyproject.toml`, dependency `script-languages-container-tools` has the same major version
+
+### Impact on NC Tests
+
+Some of the tests of the Notebook Connector, especially the integration tests, may depend on particular properties of a particular SLCR release, e.g. a particular flavor to be present.
+
+In consequence updating the SLCR version potentially may require to update the NC tests, e.g. the name of the flavor used in the tests.

doc/user_guide/user-guide.md

@@ -1,13 +1,25 @@
 # Notebook Connector User Guide
 
-## SlctManager
+## Managing Script Language Containers (SLCs)
 
-Class `SlctManager` in the Notebook Connector (NC) supports building different flavors of [Exasol Script Language Containers](https://github.com/exasol/script-languages-release) (SLCs). SlctManager uses [script-languages-container-tool](https://github.com/exasol/script-languages-container-tool) internally.
+The Notebook Connector (NC) supports building different flavors of [Exasol Script Language Containers](https://github.com/exasol/script-languages-release) (SLCs) using the [script-languages-container-tool](https://github.com/exasol/script-languages-container-tool).
 
-* The name of the SLC flavor must be provided in the Secure Configuration Storage (SCS) passed as parameter `secrets` to the constructor of SlctManager.
-* Additionally the caller must specify the *key* in the SCS for finding the flavor name.
+The specific options for building an SLC are stored in the Secure Configuration Storage (SCS).  Each SLC is identified by an arbitrary unique name used as index into the SCS for finding the related options.
 
-The constructor therefore supports the additional optional parameter `session`:
+You can set the SLC options using the class method `ScriptLanguageContainer.create()`, with parameters
+* `secrets`: The SCS
+* `name`: The name of the SLC instance
+  * will be converted to upper-case and must be unique
+* `flavor`: The name of a template as provided by the [Exasol Script Language Containers](https://github.com/exasol/script-languages-release).
 
-![](slct-manager-parameters.drawio.png)
+Method `create()` will then
+* Select a Language Alias for executing UDF scripts inside the SLC
+  * See section _Define your own script aliases_ on [docs.exasol.com](https://docs.exasol.com/db/latest/database_concepts/udf_scripts/adding_new_packages_script_languages.htm).
+  * The Language Alias will use prefix `custom_slc_` followed by the specified name
+  * Consecutive call to method `deploy()` will overwrite the SLC using the same Language Alias.
+* Save the `flavor` to the SCS indexed by the SLC's name.
+* Raise an error if the name has already been used.
+* Clone the SLC Git repository to the local file system.
+
+The constructor of class `ScriptLanguageContainer` verifies the SCS to contain the flavor and the SLC repository to be cloned to the local file system.
 

exasol/nb_connector/slc/__init__.py

@@ -0,0 +1,5 @@
+from exasol.nb_connector.slc.script_language_container import (
+    PipPackageDefinition,
+    ScriptLanguageContainer,
+)
+from exasol.nb_connector.slc.slc_flavor import SlcError

exasol/nb_connector/slc/constants.py

@@ -0,0 +1,25 @@
+from pathlib import Path
+
+from exasol.nb_connector.language_container_activation import ACTIVATION_KEY_PREFIX
+
+DEFAULT_ALIAS = "ai_lab_default"  # To be used in the Jupyter notebooks of the AI Lab
+PATH_IN_BUCKET = "container"
+
+SLC_DOCKER_IMG_NAME = "exasol/script-language-container"
+
+SLC_ACTIVATION_KEY_PREFIX = ACTIVATION_KEY_PREFIX + "slc_"
+"""
+Activation SQL for the Custom SLC will be saved in the Secure
+Configuration Storage with this key.
+"""
+
+FLAVORS_PATH_IN_SLC_REPO = Path("flavors")
+"""Path to flavors within the script-languages-release repository"""
+
+SLC_RELEASE_TAG = "9.7.0"
+"""
+Using the SLC_RELEASE 9.7.0 because we are limited to slc-tool 3.*. (see pyproject.toml)
+Check the developer guide (./doc/developer-guide.md) for more information.
+"""
+
+WORKSPACE_DIR = ".workspace"

exasol/nb_connector/slc/script_language_container.py

@@ -0,0 +1,212 @@
+from __future__ import annotations
+
+import re
+from collections import namedtuple
+from pathlib import (
+    Path,
+)
+
+from exasol.slc import api as exaslct_api
+from exasol_integration_test_docker_environment.lib.docker import (
+    ContextDockerClient,
+)
+
+from exasol.nb_connector.ai_lab_config import AILabConfig as CKey
+from exasol.nb_connector.language_container_activation import ACTIVATION_KEY_PREFIX
+from exasol.nb_connector.secret_store import Secrets
+from exasol.nb_connector.slc import constants
+from exasol.nb_connector.slc.slc_flavor import (
+    SlcError,
+    SlcFlavor,
+)
+from exasol.nb_connector.slc.workspace import (
+    Workspace,
+    current_directory,
+)
+
+PipPackageDefinition = namedtuple("PipPackageDefinition", ["pkg", "version"])
+
+
+class ScriptLanguageContainer:
+    """
+    Support building different flavors of Exasol Script Language
+    Containers (SLCs) using the SLCT.
+
+    Parameter ``name`` serves as base of the language alias and a key for the related flavor stored in the
+    Secure Configuration Storage (SCS / secrets / conf).  The flavor is used
+    as a template for building the SLC.
+
+    If the flavor is missing in the SCS or the SLC Git repository has not been
+    checked out (i.e. cloned) into the checkout_dir, then the constructor will
+    raise an SlcError.
+
+    Additionally, the caller needs to ensure, that a flavor with this name is
+    contained in the SLC release specified in variable
+    constants.SLC_RELEASE_TAG.
+    """
+
+    def __init__(
+        self,
+        secrets: Secrets,
+        name: str,
+    ):
+        self.secrets = secrets
+        self.name = name
+        self.flavor = SlcFlavor(name).verify(secrets)
+        self.workspace = Workspace.for_slc(name)
+        if not self.flavor_path.is_dir():
+            raise SlcError(
+                f"SLC Git repository not checked out to {self.checkout_dir}."
+            )
+
+    @classmethod
+    def create(
+        cls,
+        secrets: Secrets,
+        name: str,
+        flavor: str,
+    ) -> ScriptLanguageContainer:
+        slc_flavor = SlcFlavor(name)
+        if slc_flavor.exists(secrets):
+            raise SlcError(
+                "Secure Configuration Storage already contains a"
+                f" flavor for SLC name {name}."
+            )
+        slc_flavor.save(secrets, flavor)
+        workspace = Workspace.for_slc(name)
+        workspace.clone_slc_repo()
+        return cls(secrets=secrets, name=name)
+
+    @property
+    def language_alias(self) -> str:
+        """
+        Is case-insensitive.
+        """
+        return f"custom_slc_{self.name}"
+
+    @property
+    def checkout_dir(self) -> Path:
+        return self.workspace.git_clone_path
+
+    @property
+    def _flavor_path_rel(self) -> str:
+        return str(self.flavor_path.relative_to(self.checkout_dir))
+
+    @property
+    def flavor_path(self) -> Path:
+        return self.checkout_dir / constants.FLAVORS_PATH_IN_SLC_REPO / self.flavor
+
+    @property
+    def custom_pip_file(self) -> Path:
+        """
+        Returns the path to the custom pip file of the flavor
+        """
+        return (
+            self.flavor_path
+            / "flavor_customization"
+            / "packages"
+            / "python3_pip_packages"
+        )
+
+    def export(self):
+        """
+        Exports the current SLC to the export directory.
+        """
+        with current_directory(self.checkout_dir):
+            exaslct_api.export(
+                flavor_path=(str(self._flavor_path_rel),),
+                export_path=str(self.workspace.export_path),
+                output_directory=str(self.workspace.output_path),
+                release_name=self.language_alias,
+            )
+
+    def deploy(self):
+        """
+        Deploys the current script-languages-container to the database and
+        stores the activation string in the Secure Configuration Storage.
+        """
+        bfs_params = {
+            k: self.secrets.get(v)
+            for k, v in [
+                ("bucketfs_host", CKey.bfs_host_name),
+                ("bucketfs_port", CKey.bfs_port),
+                ("bucketfs_user", CKey.bfs_user),
+                ("bucketfs_password", CKey.bfs_password),
+                ("bucketfs_name", CKey.bfs_service),
+                ("bucket", CKey.bfs_bucket),
+            ]
+        }
+
+        with current_directory(self.checkout_dir):
+            exaslct_api.deploy(
+                flavor_path=(str(self._flavor_path_rel),),
+                **bfs_params,
+                path_in_bucket=constants.PATH_IN_BUCKET,
+                release_name=self.language_alias,
+                output_directory=str(self.workspace.output_path),
+            )
+            container_name = f"{self.flavor}-release-{self.language_alias}"
+            result = exaslct_api.generate_language_activation(
+                flavor_path=str(self._flavor_path_rel),
+                bucketfs_name=bfs_params["bucketfs_name"],
+                bucket_name=bfs_params["bucket"],
+                container_name=container_name,
+                path_in_bucket=constants.PATH_IN_BUCKET,
+            )
+            alter_session_cmd = result[0]
+            re_res = re.search(
+                r"ALTER SESSION SET SCRIPT_LANGUAGES='(.*)'", alter_session_cmd
+            )
+            activation_key = re_res.groups()[0]
+            _, url = activation_key.split("=", maxsplit=1)
+            self.secrets.save(self._alias_key, f"{self.language_alias}={url}")
+
+    @property
+    def _alias_key(self):
+        return constants.SLC_ACTIVATION_KEY_PREFIX + self.language_alias
+
+    @property
+    def activation_key(self) -> str:
+        """
+        Returns the language activation string for the uploaded script-language-container.
+        Can be used in `ALTER SESSION` or `ALTER_SYSTEM` SQL commands to activate
+        the language of the uploaded script-language-container.
+        """
+        try:
+            return self.secrets[self._alias_key]
+        except AttributeError as ex:
+            raise SlcError(
+                "Secure Configuration Storage does not contains an activation key."
+            ) from ex
+
+    def append_custom_packages(self, pip_packages: list[PipPackageDefinition]):
+        """
+        Appends packages to the custom pip file.
+
+        Note: This method is not idempotent: Multiple calls with the same
+        package definitions will result in duplicate entries.
+        """
+        with open(self.custom_pip_file, "a") as f:
+            for p in pip_packages:
+                print(f"{p.pkg}|{p.version}", file=f)
+
+    @property
+    def docker_image_tags(self) -> list[str]:
+        """
+        Return list of Docker image tags related to the current SLC.
+        """
+        image_name = constants.SLC_DOCKER_IMG_NAME
+        prefix = f"{image_name}:{self.flavor}"
+        with ContextDockerClient() as docker_client:
+            images = docker_client.images.list(name=image_name)
+            return [tag for img in images if (tag := img.tags[0]).startswith(prefix)]
+
+    def clean_docker_images(self):
+        """
+        Deletes local docker images related to the current SLC.
+        """
+        with current_directory(self.checkout_dir):
+            exaslct_api.clean_flavor_images(
+                flavor_path=(str(self._flavor_path_rel),),
+                output_directory=str(self.workspace.output_path),
+            )