Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
## **Solution**
This pull request introduces a new container abstraction that consolidates Docker and Kubernetes container operations under a unified interface. The changes include:
* Renaming the package from "docker" to "container"
* Introducing the following core components:
- **ContainerImpl**: Provides a unified interface to interact with container runtimes. It smartly selects between the Docker wrapper and a new Kubernetes wrapper based on the runtime environment.
- **ContainerOpts**: Encapsulates container configuration options. In contrast to the old DockerOpts (which used maps for volumes and environment variables), the new design uses a slice for volumes and a map for environment variables. This makes it easier to manage and extend configurations.
- **DockerOpts**: Now supports additional options such as restart policy and aligns with the new structure.
- **K8sOpts**: Contains options specific to running containers in a Kubernetes cluster.
* Adaptations in application code:
- In the Airbyte provider storage module, the previous `docker.DockerWrapper` has been replaced by the unified container implementation (`ContainerImpl`). This ensures that all container operations (e.g., volume mounting, environment variables, command execution) work consistently whether running in a Docker or Kubernetes environment.
- In the DBT runner within the transformer registry, similar changes have been applied by replacing references to the old Docker abstraction with the new container abstraction.
* New Kubernetes wrapper implementation
- A new wrapper (`K8sWrapper`) has been introduced. When the environment variable `KUBERNETES_SERVICE_HOST` is detected, the container implementation will instantiate the Kubernetes wrapper rather than the Docker one.
- This wrapper provides methods for creating pods, retrieving logs, and deleting pods in a Kubernetes cluster, bridging the gap between local container operations and cloud-native orchestration.
- Comprehensive testing has been added, including unit tests for the Kubernetes options and integration tests that create a temporary Kind cluster, where an actual pod is spawned to test log retrieval.
The diagrams below explain the flow and interactions of the new container abstraction:
### **1. Sequence Diagram**
**Purpose:** Illustrates the decision-making process of choosing the proper container runtime interface.
* The application creates a configuration using **ContainerOpts**.
* The helper function `NewContainerImpl(l)` checks if the environment variable `KUBERNETES_SERVICE_HOST` is set.
* If running in a Kubernetes environment, `NewK8sWrapper` is invoked to create a Kubernetes client.
* Otherwise, the system falls back to the traditional Docker wrapper via `NewDockerWrapper(l)`.
* **ContainerImpl** converts these options to the runtime-specific format and delegates to either the Docker or Kubernetes wrapper.
* The respective wrapper then pulls the image (if necessary) and runs the container/pod, returning logs and status.
```mermaid
flowchart TD
A[Create ContainerOpts]
B[NewContainerImpl]
C{Is Kubernetes Environment?}
D[NewK8sWrapper]
E[NewDockerWrapper]
F[Run Container/Pod]
G[Return Logs & Status]
A --> B
B --> C
C -- Yes --> D
C -- No --> E
D --> F
E --> F
F --> G
```
### **2. Class Diagram**
**Purpose:** Shows the relationships between key components managing container operations.
* The central **ContainerImpl** interface defines common methods such as Run() and Pull().
* The **DockerWrapper** and **K8sWrapper** implement the ContainerImpl interface.
* **ContainerOpts** encapsulates configuration shared between the two implementations.
* **K8sOpts** and **DockerOpts** serve as the runtime-specific configuration options.
```mermaid
classDiagram
class ContainerImpl {
+Run(ctx, opts ContainerOpts) (stdout io.Reader, stderr io.Reader, err error)
+Pull(ctx, image string, opts ImagePullOptions) error
}
class DockerWrapper {
+RunContainer(ctx, opts DockerOpts) (io.Reader, io.Reader, error)
+Pull(ctx, image string, opts ImagePullOptions) error
}
class K8sWrapper {
+Run(ctx, opts ContainerOpts) (io.Reader, io.Reader, error)
+RunPod(ctx, opts K8sOpts) (*bytes.Buffer, error)
+Pull(ctx, image string, opts ImagePullOptions) error
}
class ContainerOpts {
+Env map[string]string
+Volumes []Volume
+LogDriver string
+LogOptions map[string]string
+Image string
+Network string
+ContainerName string
+Command []string
+Args []string
+Timeout time.Duration
+AttachStdout bool
+AttachStderr bool
+RestartPolicy string
}
class DockerOpts
class K8sOpts
ContainerImpl <|.. DockerWrapper
ContainerImpl <|.. K8sWrapper
ContainerOpts --> DockerOpts : convertTo()
ContainerOpts --> K8sOpts : convertTo()
```
Let me know if further refinements are needed!
---
Pull Request resolved: #206
Co-authored-by: tserakhau <tserakhau@double.cloud>
Co-authored-by: tserakhau <tserakhau@double.cloud>
Co-authored-by: tserakhau <tserakhau@double.cloud>
Co-authored-by: tserakhau <tserakhau@double.cloud>
Co-authored-by: tserakhau <tserakhau@double.cloud>
Co-authored-by: tserakhau <tserakhau@double.cloud>
commit_hash:19988b16b4a75ec523f1e2476a60c890837066a0- Loading branch information
1 parent
173effe
commit fd08e3c