-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Utility for private registries and installer image rootless (#48)
- Loading branch information
1 parent
d1a6e3a
commit 4760b2a
Showing
14 changed files
with
238 additions
and
41 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| ## Codefresh gitops runtime | ||
| {{ template "chart.versionBadge" . }}{{ template "chart.typeBadge" . }}{{ template "chart.appVersionBadge" . }} | ||
|
|
||
| ## Codefresh official documentation: | ||
| Prior to running the installation please see the official documentation at: https://codefresh.io/docs/docs/installation/gitops/hybrid-gitops-helm-installation/ | ||
|
|
||
| ## Using with private registries - Helper utility | ||
| The GitOps Runtime comprises multiple subcharts and container images. Subcharts also vary in values structure, making it difficult to override image specific values to use private registries. | ||
| We have created a helper utility to resolve this issue: | ||
| - The utility create values files in the correct structure, overriding the registry for each image. When installing the chart, you can then provide those values files to override all images. | ||
| - The utility also creates other files with data to help you identify and correctly mirror all the images. | ||
|
|
||
| #### Usage | ||
|
|
||
| The utility is packaged in a container image. Below are instructions on executing the utility using Docker: | ||
|
|
||
| ``` | ||
| docker run -v <output_dir>:/output quay.io/codefresh/gitops-runtime-private-registry-utils:0.2.7-alpha <local_registry> | ||
| ``` | ||
| `output_dir` - is a local directory where the utility will output files. <br> | ||
| `local_registry` - is your local registry where you want to mirror the images to | ||
|
|
||
| The utility will output 4 files into the folder: | ||
| 1. `image-list.txt` - is the list of all images used in this version of the chart. Those are the images that you need to mirror. | ||
| 2. `image-mirror.csv` - is a csv file with 2 fields - source_image and target_image. source_image is the image with the original registry and target_image is the image with the private registry. Can be used as an input file for a mirroring script. | ||
| 3. `values-images-no-tags.yaml` - a values file with all image values with the private registry **excluding tags**. If provided through --values to helm install/upgrade command - it will override all images to use the private registry. | ||
| 4. `values-images-with-tags.yaml` - The same as 3 but with tags **included**. | ||
|
|
||
| {{ template "chart.valuesSection" . }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| README.md | ||
| Dockerfile |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| FROM --platform=$BUILDPLATFORM python:3.11.3-slim-bullseye | ||
| ARG TARGETARCH | ||
| RUN cd /tmp && python3 -c "from urllib.request import urlretrieve; urlretrieve('https://get.helm.sh/helm-v3.12.0-linux-${TARGETARCH}.tar.gz', 'helm-v3.12.0-linux-${TARGETARCH}.tar.gz')" && tar -xvf helm-v3.12.0-linux-${TARGETARCH}.tar.gz && chmod +x linux-${TARGETARCH}/helm && mv linux-${TARGETARCH}/helm /usr/local/bin/helm && rm -rf /tmp/* | ||
| COPY charts/gitops-runtime /chart | ||
| RUN helm dependency update /chart | ||
| COPY scripts/private-registry-utils/python-requirements.txt /scripts/python-requirements.txt | ||
| RUN pip3 install -r /scripts/python-requirements.txt | ||
| COPY scripts/private-registry-utils /scripts | ||
| RUN chmod -R +x /scripts | ||
| WORKDIR /scripts | ||
| # Output calculated values and filter image values | ||
| RUN ./output-calculated-values.sh ./all-values.yaml && python3 ./helper-scripts/yaml-filter.py all-values.yaml image.repository,image.registry,image.tag,argo-events.configs.nats.versions,argo-events.configs.jetstream.versions > all-image-values.yaml | ||
| ENTRYPOINT ["python3", "private-registry-utils.py", "all-image-values.yaml"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| Utilities to assist with image mirroring to private registries | ||
|
|
||
| ## How it works? | ||
|
|
||
| 1. All calculated values are outputed using output-calculated-values.sh. The script contains a special template that handles image values. Specifically it derives tags for images that don't have explicit tags sepcified from the subchart appVersion. | ||
| 2. Those values are then filtered using the helper script yaml-filter.py to filter only the values that contain images. | ||
| 3. image-values-utils.py then uses the output of the previous steps and can generate: | ||
| 1. A list of all images | ||
| 2. Values files with and without tags including a custom registry for each image. | ||
| 3. A csv file with source and target image names (can help with image mirroring). | ||
|
|
||
| ## Usage | ||
| 1. Build the image with buildcontext being the root of the repo `docker build -f scripts/private-registry-utils/Dockerfile -t <image-tag> .` For example: `docker build -f scripts/private-registry-utils/Dockerfile -t gitops-runtime-priv-reg .` | ||
| 2. Execute with: `docker run -v <local output directory>:/output -it <image-tag> <private registry>` for example: `docker run -v /tmp/output:/output -it gitops-runtime-priv-reg myregisry.example.com` | ||
| 3. See the generated files in local output folder |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
150 changes: 150 additions & 0 deletions
150
scripts/private-registry-utils/private-registry-utils.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,150 @@ | ||
| import argparse | ||
| import yaml | ||
| import sys | ||
| import re | ||
| import csv | ||
|
|
||
| DEFAULT_REGISTRY_URL = "registry.example.com" | ||
|
|
||
| def remove_duplicates(list_of_dicts, key): | ||
| seen = set() | ||
| deduplicated_list = [] | ||
|
|
||
| for d in list_of_dicts: | ||
| if d[key] not in seen: | ||
| deduplicated_list.append(d) | ||
| seen.add(d[key]) | ||
|
|
||
| return deduplicated_list | ||
|
|
||
| def replace_registry_in_image(image_string, new_registry): | ||
| if '/' in image_string: | ||
| parts = image_string.split('/') | ||
| if len(parts) >= 2: | ||
| parts[0] = new_registry | ||
| return '/'.join(parts) | ||
| else: | ||
| return new_registry + '/' + image_string | ||
|
|
||
| # Try to identify whether a string is a docker image | ||
| def is_docker_image(image_string): | ||
| pattern = r'^[a-zA-Z0-9/_-]+:[a-zA-Z0-9_.-]+$' | ||
| return bool(re.match(pattern, image_string)) | ||
|
|
||
|
|
||
| def recurse_replace_registry(currValue,new_registry): | ||
| if type(currValue) is dict: | ||
| for key in currValue.keys(): | ||
| if key == "registry": | ||
| currValue[key]=new_registry | ||
| elif key == "repository" and "registry" not in currValue: | ||
| currValue[key] = replace_registry_in_image(currValue[key],new_registry) | ||
| elif type(currValue[key]) is str: | ||
| if is_docker_image(currValue[key]): | ||
| currValue[key] = replace_registry_in_image(currValue[key],new_registry) | ||
| else: | ||
| recurse_replace_registry(currValue[key],new_registry) | ||
| elif type(currValue) is list: | ||
| for item in currValue: | ||
| recurse_replace_registry(item,new_registry) | ||
|
|
||
| def recurse_remove_tags(currValue): | ||
| if type(currValue) is dict: | ||
| if "tag" in currValue: | ||
| currValue.pop("tag") | ||
| else: | ||
| for key in currValue.keys(): | ||
| recurse_remove_tags(currValue[key]) | ||
| elif type(currValue) is list: | ||
| for item in currValue: | ||
| recurse_remove_tags(item) | ||
|
|
||
|
|
||
| def recurse_get_source_target(currValue,new_registry,lstSourceTarget): | ||
| sourceImage = "" | ||
| if type(currValue) is dict: | ||
| for key in currValue.keys(): | ||
| if key == "registry": | ||
| sourceImage += currValue[key] + "/" | ||
| if key == "repository": | ||
| sourceImage += currValue[key] | ||
| if key == "tag": | ||
| sourceImage += ":" + currValue[key] | ||
|
|
||
| elif type(currValue[key]) is str: | ||
| if is_docker_image(currValue[key]): | ||
| sourceImage = currValue[key] | ||
|
|
||
| recurse_get_source_target(currValue[key],new_registry,lstSourceTarget) | ||
|
|
||
| if len(sourceImage) > 0: | ||
| lstSourceTarget.append({"source_image": sourceImage, "target_image": replace_registry_in_image(sourceImage,new_registry)}) | ||
|
|
||
|
|
||
| elif type(currValue) is list: | ||
| for item in currValue: | ||
| recurse_get_source_target(item,new_registry,lstSourceTarget) | ||
|
|
||
| def generate_file_from_field(list_of_dicts, field_name, output_file): | ||
| with open(output_file, 'w+') as file: | ||
| for d in list_of_dicts: | ||
| field_value = d.get(field_name) | ||
| if field_value: | ||
| file.write(str(field_value) + '\n') | ||
|
|
||
| def generate_image_values(dictImageValues,outputDir): | ||
|
|
||
| with open(f"{outputDir}/values-images-with-tags.yaml", 'w+') as file: | ||
| yaml.dump(dictImageValues, file) | ||
|
|
||
| recurse_remove_tags(dictImageValues) | ||
|
|
||
| with open(f"{outputDir}/values-images-no-tags.yaml", 'w+') as file: | ||
| yaml.dump(dictImageValues, file) | ||
|
|
||
| def generate_image_list(): | ||
| # Code for generating image list | ||
| print("Generating image list") | ||
|
|
||
| def generate_mirror_csv(lstSourceTarget, outputDir): | ||
|
|
||
| fields = ['source_image', 'target_image'] | ||
|
|
||
| with open(f"{outputDir}/image-mirror.csv", 'w+') as csvfile: | ||
| writer = csv.DictWriter(csvfile, fieldnames = fields) | ||
| writer.writeheader() | ||
| writer.writerows(lstSourceTarget) | ||
|
|
||
| def main(): | ||
|
|
||
| parser = argparse.ArgumentParser(description="Codefresh gitops runtime - private registry utils") | ||
| parser.add_argument("images_values_file", help="Input values.yaml file") | ||
| parser.add_argument("private_registry_url", nargs="?", default=DEFAULT_REGISTRY_URL, help="Private Registry URL") | ||
| parser.add_argument("--output-dir", help="Output directory",default="/output") | ||
| parser.add_argument("--action", choices=["generate-image-values", "generate-image-list", "generate-mirror-csv"], help="Action to execute") | ||
| args = parser.parse_args() | ||
|
|
||
| # Open yaml | ||
| with open(args.images_values_file, 'r') as stream: | ||
| try: | ||
| dictImageValues=yaml.safe_load(stream) | ||
| except yaml.YAMLError as e: | ||
| print(e) | ||
|
|
||
| lstSourceTarget = [] | ||
| recurse_get_source_target(dictImageValues,args.private_registry_url,lstSourceTarget) | ||
| lstSourceTarget = remove_duplicates(lstSourceTarget, "source_image") | ||
| recurse_replace_registry(dictImageValues,args.private_registry_url) | ||
|
|
||
|
|
||
| if args.action == "generate-mirror-csv" or args.action is None: | ||
| generate_mirror_csv(lstSourceTarget,args.output_dir) | ||
|
|
||
| if args.action == "generate-image-list" or args.action is None: | ||
| generate_file_from_field(lstSourceTarget,"source_image", f"{args.output_dir}/image-list.txt") | ||
|
|
||
| if args.action == "generate-image-values" or args.action is None: | ||
| generate_image_values(dictImageValues,args.output_dir) | ||
|
|
||
| if __name__ == "__main__": | ||
| main() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| PyYAML==6.0 |