Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kops: test alternative registry for periodics-distros #25155

Merged
merged 1 commit into from
Feb 9, 2022
Merged

kops: test alternative registry for periodics-distros #25155

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion config/jobs/kubernetes/kops/build_jobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -747,11 +747,12 @@ def generate_distros():
distro_short = distro.replace('ubuntu', 'u').replace('debian', 'deb').replace('amazonlinux', 'amzn') # pylint: disable=line-too-long
results.append(
build_test(distro=distro_short,
networking='calico',
networking='kubenet',
k8s_version='stable',
kops_channel='alpha',
name_override=f"kops-aws-distro-image{distro}",
extra_dashboards=['kops-distros'],
extra_flags=['--set=spec.assets.containerProxy=registry-sandbox.k8s.io'],
Copy link
Member

@hakman hakman Feb 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will require networking='kubenet' to work, as any Calico assets will b missing from sandbox.k8s.io, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only assets concerned are the kubernetes container images from k8s.gcr.io. PR updated.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but this setting affects all assets, not just the k8s.gcr.io ones. Thanks for updating.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hakman What's the best approach if I want to extend this change to other periodic jobs ? Is this change enough or do I need to make changes per use ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most jobs use CNIs (as kubenet style networking is not really a production use case). This means that setting this proxy will not work for that case, unless the proxy is made to proxy CNI repos too.
Most likely path is to decide on a final dns name for the CDN cached registry and switch all kOps manifests to use it during the kOps 1.24 development cycle. We can decide later during that cycle if we want to keep it or revert to k8s.gcr.io.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the help!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My pleasure. Thanks for trying to improve this. I think it will become a huge improvement for many k8s users in the future if successful.

runs_per_day=3,
)
)
Expand Down
64 changes: 32 additions & 32 deletions config/jobs/kubernetes/kops/kops-periodics-distros.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# 8 jobs, total of 168 runs per week
periodics:

# {"cloud": "aws", "container_runtime": "containerd", "distro": "deb10", "extra_flags": "--discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "calico"}
# {"cloud": "aws", "container_runtime": "containerd", "distro": "deb10", "extra_flags": "--set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "kubenet"}
- name: e2e-kops-aws-distro-imagedebian10
cron: '30 6-23/8 * * *'
labels:
Expand Down Expand Up @@ -31,7 +31,7 @@ periodics:
-v 2 \
--up --down \
--cloud-provider=aws \
--create-args="--image='136693071363/debian-10-amd64-20211011-792' --channel=alpha --networking=calico --container-runtime=containerd --discovery-store=s3://k8s-kops-prow/discovery" \
--create-args="--image='136693071363/debian-10-amd64-20211011-792' --channel=alpha --networking=kubenet --container-runtime=containerd --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery" \
--kops-version-marker=https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt \
--kubernetes-version=https://storage.googleapis.com/kubernetes-release/release/stable.txt \
--test=kops \
Expand All @@ -57,16 +57,16 @@ periodics:
test.kops.k8s.io/cloud: aws
test.kops.k8s.io/container_runtime: containerd
test.kops.k8s.io/distro: deb10
test.kops.k8s.io/extra_flags: --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/extra_flags: --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/k8s_version: stable
test.kops.k8s.io/kops_channel: alpha
test.kops.k8s.io/kops_version: ''
test.kops.k8s.io/networking: calico
test.kops.k8s.io/networking: kubenet
testgrid-dashboards: google-aws, kops-distro-deb10, kops-distros, kops-k8s-stable, kops-latest, sig-cluster-lifecycle-kops
testgrid-days-of-results: '90'
testgrid-tab-name: kops-aws-distro-imagedebian10

# {"cloud": "aws", "container_runtime": "containerd", "distro": "deb11", "extra_flags": "--discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "calico"}
# {"cloud": "aws", "container_runtime": "containerd", "distro": "deb11", "extra_flags": "--set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "kubenet"}
- name: e2e-kops-aws-distro-imagedebian11
cron: '48 0-23/8 * * *'
labels:
Expand Down Expand Up @@ -95,7 +95,7 @@ periodics:
-v 2 \
--up --down \
--cloud-provider=aws \
--create-args="--image='136693071363/debian-11-amd64-20220121-894' --channel=alpha --networking=calico --container-runtime=containerd --discovery-store=s3://k8s-kops-prow/discovery" \
--create-args="--image='136693071363/debian-11-amd64-20220121-894' --channel=alpha --networking=kubenet --container-runtime=containerd --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery" \
--kops-version-marker=https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt \
--kubernetes-version=https://storage.googleapis.com/kubernetes-release/release/stable.txt \
--test=kops \
Expand All @@ -121,16 +121,16 @@ periodics:
test.kops.k8s.io/cloud: aws
test.kops.k8s.io/container_runtime: containerd
test.kops.k8s.io/distro: deb11
test.kops.k8s.io/extra_flags: --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/extra_flags: --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/k8s_version: stable
test.kops.k8s.io/kops_channel: alpha
test.kops.k8s.io/kops_version: ''
test.kops.k8s.io/networking: calico
test.kops.k8s.io/networking: kubenet
testgrid-dashboards: google-aws, kops-distro-deb11, kops-distros, kops-k8s-stable, kops-latest, sig-cluster-lifecycle-kops
testgrid-days-of-results: '90'
testgrid-tab-name: kops-aws-distro-imagedebian11

# {"cloud": "aws", "container_runtime": "containerd", "distro": "u2004", "extra_flags": "--discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "calico"}
# {"cloud": "aws", "container_runtime": "containerd", "distro": "u2004", "extra_flags": "--set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "kubenet"}
- name: e2e-kops-aws-distro-imageubuntu2004
cron: '53 6-23/8 * * *'
labels:
Expand Down Expand Up @@ -159,7 +159,7 @@ periodics:
-v 2 \
--up --down \
--cloud-provider=aws \
--create-args="--image='099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20220131' --channel=alpha --networking=calico --container-runtime=containerd --discovery-store=s3://k8s-kops-prow/discovery" \
--create-args="--image='099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20220131' --channel=alpha --networking=kubenet --container-runtime=containerd --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery" \
--kops-version-marker=https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt \
--kubernetes-version=https://storage.googleapis.com/kubernetes-release/release/stable.txt \
--test=kops \
Expand All @@ -185,16 +185,16 @@ periodics:
test.kops.k8s.io/cloud: aws
test.kops.k8s.io/container_runtime: containerd
test.kops.k8s.io/distro: u2004
test.kops.k8s.io/extra_flags: --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/extra_flags: --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/k8s_version: stable
test.kops.k8s.io/kops_channel: alpha
test.kops.k8s.io/kops_version: ''
test.kops.k8s.io/networking: calico
test.kops.k8s.io/networking: kubenet
testgrid-dashboards: google-aws, kops-distro-u2004, kops-distros, kops-k8s-stable, kops-latest, sig-cluster-lifecycle-kops
testgrid-days-of-results: '90'
testgrid-tab-name: kops-aws-distro-imageubuntu2004

# {"cloud": "aws", "container_runtime": "containerd", "distro": "u2110", "extra_flags": "--discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "calico"}
# {"cloud": "aws", "container_runtime": "containerd", "distro": "u2110", "extra_flags": "--set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "kubenet"}
- name: e2e-kops-aws-distro-imageubuntu2110
cron: '14 1-23/8 * * *'
labels:
Expand Down Expand Up @@ -223,7 +223,7 @@ periodics:
-v 2 \
--up --down \
--cloud-provider=aws \
--create-args="--image='099720109477/ubuntu/images/hvm-ssd/ubuntu-impish-21.10-amd64-server-20220201' --channel=alpha --networking=calico --container-runtime=containerd --discovery-store=s3://k8s-kops-prow/discovery" \
--create-args="--image='099720109477/ubuntu/images/hvm-ssd/ubuntu-impish-21.10-amd64-server-20220201' --channel=alpha --networking=kubenet --container-runtime=containerd --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery" \
--kops-version-marker=https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt \
--kubernetes-version=https://storage.googleapis.com/kubernetes-release/release/stable.txt \
--test=kops \
Expand All @@ -249,16 +249,16 @@ periodics:
test.kops.k8s.io/cloud: aws
test.kops.k8s.io/container_runtime: containerd
test.kops.k8s.io/distro: u2110
test.kops.k8s.io/extra_flags: --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/extra_flags: --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/k8s_version: stable
test.kops.k8s.io/kops_channel: alpha
test.kops.k8s.io/kops_version: ''
test.kops.k8s.io/networking: calico
test.kops.k8s.io/networking: kubenet
testgrid-dashboards: google-aws, kops-distro-u2110, kops-distros, kops-k8s-stable, kops-latest, sig-cluster-lifecycle-kops
testgrid-days-of-results: '90'
testgrid-tab-name: kops-aws-distro-imageubuntu2110

# {"cloud": "aws", "container_runtime": "containerd", "distro": "u2204", "extra_flags": "--discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "calico"}
# {"cloud": "aws", "container_runtime": "containerd", "distro": "u2204", "extra_flags": "--set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "kubenet"}
- name: e2e-kops-aws-distro-imageubuntu2204
cron: '47 0-23/8 * * *'
labels:
Expand Down Expand Up @@ -287,7 +287,7 @@ periodics:
-v 2 \
--up --down \
--cloud-provider=aws \
--create-args="--image='099720109477/ubuntu/images-testing/hvm-ssd/ubuntu-jammy-daily-amd64-server-20220204' --channel=alpha --networking=calico --container-runtime=containerd --discovery-store=s3://k8s-kops-prow/discovery" \
--create-args="--image='099720109477/ubuntu/images-testing/hvm-ssd/ubuntu-jammy-daily-amd64-server-20220204' --channel=alpha --networking=kubenet --container-runtime=containerd --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery" \
--kops-version-marker=https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt \
--kubernetes-version=https://storage.googleapis.com/kubernetes-release/release/stable.txt \
--test=kops \
Expand All @@ -313,16 +313,16 @@ periodics:
test.kops.k8s.io/cloud: aws
test.kops.k8s.io/container_runtime: containerd
test.kops.k8s.io/distro: u2204
test.kops.k8s.io/extra_flags: --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/extra_flags: --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/k8s_version: stable
test.kops.k8s.io/kops_channel: alpha
test.kops.k8s.io/kops_version: ''
test.kops.k8s.io/networking: calico
test.kops.k8s.io/networking: kubenet
testgrid-dashboards: google-aws, kops-distro-u2204, kops-distros, kops-k8s-stable, kops-latest, sig-cluster-lifecycle-kops
testgrid-days-of-results: '90'
testgrid-tab-name: kops-aws-distro-imageubuntu2204

# {"cloud": "aws", "container_runtime": "containerd", "distro": "amzn2", "extra_flags": "--discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "calico"}
# {"cloud": "aws", "container_runtime": "containerd", "distro": "amzn2", "extra_flags": "--set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "kubenet"}
- name: e2e-kops-aws-distro-imageamazonlinux2
cron: '55 3-23/8 * * *'
labels:
Expand Down Expand Up @@ -351,7 +351,7 @@ periodics:
-v 2 \
--up --down \
--cloud-provider=aws \
--create-args="--image='amazon/amzn2-ami-kernel-5.10-hvm-2.0.20220121.0-x86_64-gp2' --channel=alpha --networking=calico --container-runtime=containerd --discovery-store=s3://k8s-kops-prow/discovery" \
--create-args="--image='amazon/amzn2-ami-kernel-5.10-hvm-2.0.20220207.1-x86_64-gp2' --channel=alpha --networking=kubenet --container-runtime=containerd --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery" \
--kops-version-marker=https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt \
--kubernetes-version=https://storage.googleapis.com/kubernetes-release/release/stable.txt \
--test=kops \
Expand All @@ -377,16 +377,16 @@ periodics:
test.kops.k8s.io/cloud: aws
test.kops.k8s.io/container_runtime: containerd
test.kops.k8s.io/distro: amzn2
test.kops.k8s.io/extra_flags: --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/extra_flags: --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/k8s_version: stable
test.kops.k8s.io/kops_channel: alpha
test.kops.k8s.io/kops_version: ''
test.kops.k8s.io/networking: calico
test.kops.k8s.io/networking: kubenet
testgrid-dashboards: google-aws, kops-distro-amzn2, kops-distros, kops-k8s-stable, kops-latest, sig-cluster-lifecycle-kops
testgrid-days-of-results: '90'
testgrid-tab-name: kops-aws-distro-imageamazonlinux2

# {"cloud": "aws", "container_runtime": "containerd", "distro": "rhel8", "extra_flags": "--discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "calico"}
# {"cloud": "aws", "container_runtime": "containerd", "distro": "rhel8", "extra_flags": "--set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "kubenet"}
- name: e2e-kops-aws-distro-imagerhel8
cron: '49 7-23/8 * * *'
labels:
Expand Down Expand Up @@ -415,7 +415,7 @@ periodics:
-v 2 \
--up --down \
--cloud-provider=aws \
--create-args="--image='309956199498/RHEL-8.5.0_HVM-20211103-x86_64-0-Hourly2-GP2' --channel=alpha --networking=calico --container-runtime=containerd --discovery-store=s3://k8s-kops-prow/discovery" \
--create-args="--image='309956199498/RHEL-8.5.0_HVM-20211103-x86_64-0-Hourly2-GP2' --channel=alpha --networking=kubenet --container-runtime=containerd --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery" \
--kops-version-marker=https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt \
--kubernetes-version=https://storage.googleapis.com/kubernetes-release/release/stable.txt \
--test=kops \
Expand All @@ -441,16 +441,16 @@ periodics:
test.kops.k8s.io/cloud: aws
test.kops.k8s.io/container_runtime: containerd
test.kops.k8s.io/distro: rhel8
test.kops.k8s.io/extra_flags: --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/extra_flags: --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/k8s_version: stable
test.kops.k8s.io/kops_channel: alpha
test.kops.k8s.io/kops_version: ''
test.kops.k8s.io/networking: calico
test.kops.k8s.io/networking: kubenet
testgrid-dashboards: google-aws, kops-distro-rhel8, kops-distros, kops-k8s-stable, kops-latest, sig-cluster-lifecycle-kops
testgrid-days-of-results: '90'
testgrid-tab-name: kops-aws-distro-imagerhel8

# {"cloud": "aws", "container_runtime": "containerd", "distro": "flatcar", "extra_flags": "--discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "calico"}
# {"cloud": "aws", "container_runtime": "containerd", "distro": "flatcar", "extra_flags": "--set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery", "k8s_version": "stable", "kops_channel": "alpha", "kops_version": null, "networking": "kubenet"}
- name: e2e-kops-aws-distro-imageflatcar
cron: '36 6-23/8 * * *'
labels:
Expand Down Expand Up @@ -479,7 +479,7 @@ periodics:
-v 2 \
--up --down \
--cloud-provider=aws \
--create-args="--image='075585003325/Flatcar-stable-3033.2.1-hvm' --channel=alpha --networking=calico --container-runtime=containerd --discovery-store=s3://k8s-kops-prow/discovery" \
--create-args="--image='075585003325/Flatcar-stable-3033.2.1-hvm' --channel=alpha --networking=kubenet --container-runtime=containerd --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery" \
--control-plane-instance-group-overrides="spec.instanceMetadata.httpTokens=optional" \
--node-instance-group-overrides="spec.instanceMetadata.httpTokens=optional" \
--kops-version-marker=https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt \
Expand Down Expand Up @@ -508,11 +508,11 @@ periodics:
test.kops.k8s.io/cloud: aws
test.kops.k8s.io/container_runtime: containerd
test.kops.k8s.io/distro: flatcar
test.kops.k8s.io/extra_flags: --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/extra_flags: --set=spec.assets.containerProxy=registry-sandbox.k8s.io --discovery-store=s3://k8s-kops-prow/discovery
test.kops.k8s.io/k8s_version: stable
test.kops.k8s.io/kops_channel: alpha
test.kops.k8s.io/kops_version: ''
test.kops.k8s.io/networking: calico
test.kops.k8s.io/networking: kubenet
testgrid-dashboards: google-aws, kops-distro-flatcar, kops-distros, kops-k8s-stable, kops-latest, sig-cluster-lifecycle-kops
testgrid-days-of-results: '90'
testgrid-tab-name: kops-aws-distro-imageflatcar