backfill images from google-containers

[listx]

This is part of
https://github.com/kubernetes/k8s.io/blob/master/k8s.gcr.io/Vanity-Domain-Flip.md,
in preparation for the domain flip. Backfilling all images to the
top-level (root) directory ensures that the transition for existing
images post-flip will be painless.

Holding for now because I'm curious to see what the presubmit checks have to say.

/hold
/cc @thockin @justaugustus @tpepper

[listx]

I'm surprised the pull-k8sio-cip check finished so quickly (only took just over a minute).

The dry run finished without an error (I downloaded the logs and ignore-case-grepped for "fail" and "error" and found nothing), so, I think it's good for merge.

[listx]

As we get new images into google-containers, we'll have to update this manifest list.

I assume legacy-backfill is a good enough name for the directories --- but it's something we can change in a follow-up PR. Also, it's possible that we slice & dice the rather long (52K lines...!) images.yaml file into smaller chunks. Probably the first subset to move out to a separate manifest/image pair is the set of images listed on https://github.com/kubernetes/sig-release/blob/master/release-engineering/artifacts.md#docker-images for the Kubernetes releases. @justaugustus ^

[justaugustus]

Probably the first subset to move out to a separate manifest/image pair is the set of images listed on https://github.com/kubernetes/sig-release/blob/master/release-engineering/artifacts.md#docker-images for the Kubernetes releases.

@listx -- Yes, please. It'd be great to have those first-class images in their own config file; but agreed that that can be a follow-up.

[justaugustus]

For SIG Release:
/lgtm
/approve

[listx]

I had to update one of the tags in the images.yaml to reflect a manual change I just performed for cadvisor:latest. I guess I should have just added a commit on top so you can see the diff --- oh well, too late.

@justaugustus you can generate the EXACT same images.yaml file as in this PR with:

cd
git clone https://github.com/kubernetes-sigs/k8s-container-image-promoter go/src/sigs.k8s.io/k8s-container-image-promoter
cd go/src/sigs.k8s.io/k8s-container-image-promoter
bazel run \
    --workspace_status_command=$PWD/workspace_status.sh \
    --host_force_python=PY2 //:cip -- -snapshot=gcr.io/google-containers \
    -output-format=YAML > images.yaml

Can you do the above as a sanity check?

[justaugustus]

For comparison: https://gist.github.com/justaugustus/2200ff7e9da826e3a84859fe9947edd6

[justaugustus]

A diff -u between my locally generated copy and this PR, comes up empty, so this should be clear to merge, I think.

[listx]

/hold cancel

[listx]

/hold

[justaugustus]

/lgtm
/approve
/hold for Tim's review
/assign @thockin

[thockin]

Can we please get a comment in each file, and a README.md in each directory explaining that these images are effectively frozen, new changes to them will be rejected, and any new promotions MUST happen through individual sub-project staging repos?

Specifically, I want there to be NO AMBIGUITY about whether we want PRs against these, and when the next wave of community maintainers step up, they don't lose all the context.

[listx]

Dropped .gitignores, but also added a "DO-NOT-MODIFY" prefix to the folder names to be that much more explicit.

[thockin]

Will this kick off a giant import?

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justaugustus, listx, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justaugustus]

Just to weave the tracking together a little...
ref: kubernetes/release#270

[listx]

Will this kick off a giant import?

Yes. It will be interesting to see how ~50K images actually get copied over.

[thockin]

Make it so!

…

On Tue, Mar 3, 2020 at 4:29 PM Linus Arver ***@***.***> wrote: Will this kick off a giant import? Yes. It will be interesting to see how ~50K images actually get copied over. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.

[justaugustus]

...and away we go!
/hold cancel

[thockin]

I was about to make a joke about pushing to prod at 5pm, but it merged....here we go....

…

On Tue, Mar 3, 2020 at 4:49 PM Kubernetes Prow Robot < ***@***.***> wrote: Merged #623 <#623> into master. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#623?email_source=notifications&email_token=ABKWAVFKPJVWAY7Z7ZADJMTRFWQS3A5CNFSM4K76DGRKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOXBY5X5A#event-3094469620>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVDJRHHX7AKBZAUGD23RFWQS3ANCNFSM4K76DGRA> .

[listx]

It is running here: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/post-k8sio-cip/1235004600633790464

[listx]

It is still chugging along after the 1h mark now. Interestingly, the Prow job logs have been truncated (it now says only 387~ lines of logs, whereas I expect MBs of logs). @cjwagner I guess super long Prow jobs' logs getting truncated is a known (perhaps on-purpose by design) issue?

[listx]

The job has been killed by Prow because it exceeded the 2h limit. I'm pasting all my findings below, though, because I have some thoughts I should collect: https://docs.google.com/document/d/1R7RhuKwOUHsi4i3RMpaSPYLJCv16EazSww5kUQXRPlk/edit?usp=sharing

TL;DR is that I really have to fix kubernetes-sigs/promo-tools#185.

[thockin]

Cool failure mode. Did we get a clean signal that it failed? Will a periodic run pick up the debris and carry on cleanly? Or does a human have to intervene?

…

On Tue, Mar 3, 2020, 8:25 PM Linus Arver ***@***.***> wrote: The job has been killed by Prow because it exceeded the 2h limit. I'm pasting all my findings below, though, because I have some thoughts I should collect: https://docs.google.com/document/d/1R7RhuKwOUHsi4i3RMpaSPYLJCv16EazSww5kUQXRPlk/edit?usp=sharing TL;DR is that I really have to fix kubernetes-sigs/promo-tools#185 <kubernetes-sigs/promo-tools#185> . — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#623?email_source=notifications&email_token=ABKWAVFSBAM5TQTCCYQN46LRFXJ35A5CNFSM4K76DGRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENWJR2A#issuecomment-594319592>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVHVWQOOKS4WYE6XGUTRFXJ35ANCNFSM4K76DGRA> .

[cjwagner]

The job has been killed by Prow because it exceeded the 2h limit.

Note that this is just the default timeout:
https://github.com/kubernetes/test-infra/blob/268aa2ea35ddce4fa2696bb7f958327540e051b4/config/prow/config.yaml#L10

You can configure the job to have a different timeout like this: https://github.com/kubernetes/test-infra/blob/28c7418a8c8145a2b0203591b4a69262804e283a/config/jobs/kubernetes/kops/kops-presubmits.yaml#L13-L14

[listx]

Cool failure mode. Did we get a clean signal that it failed?
Yes: https://prow.k8s.io/job-history/kubernetes-jenkins/logs/post-k8sio-cip

Will a periodic run pick up the debris and carry on cleanly? Or does a human have to intervene?

Yes. The promoter is designed to only perform required promotions.

That being said I should open a PR to make the periodic job run more frequently than once a day.

[listx]

OK so the promoter is hitting an index-out-of-range error from the scheduled (daily) ci-k8sio-cip job run: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-k8sio-cip/1235320791898263552

I can reproduce locally. Will dig to see which commit in this repo caused it (I'm assuming that this PR itself did not cause this issue, but I'll have to see).

[listx]

So for some reason the read is failing because of this merge. Reverting this PR to unblock the promoter for possible other image promotions while I work on a fix.

Revert PR here: #629

[listx]

The error is happening because of a parsing failure in the function SplitByKnownRegistries. See more details here: kubernetes-sigs/promo-tools#188