[Flake] kaniko flake (5xx) fetching image from production.cloudflare.docker.com #6438

devjgm · 2021-05-03T15:25:33Z

Log: https://pantheon.corp.google.com/cloud-build/builds;region=global/a7edfd7e-c43e-43c2-a3d6-3ddba4565084;step=0?project=cloud-cpp-testing-resources

Digest: sha256:a12a027e1d0afbeb6cc31bb07e89d94dc47fa768265416350d442d878bdb6064
Status: Downloaded newer image for gcr.io/kaniko-project/executor:edge
gcr.io/kaniko-project/executor:edge
INFO[0000] Resolved base name centos:7 to devtools      
INFO[0000] Using dockerignore file: /workspace/ci/.dockerignore 
INFO[0000] Retrieving image manifest centos:7           
INFO[0000] Retrieving image manifest centos:7           
error building image: GET https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/86/8652b9f0cb4c0599575e5a003f5906876e10c1ceb2ab9fe1786712dac14a50cf/data?verify=REDACTED: unsupported status code 503; body: <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Temporarily unavailable | production.cloudflare.docker.com | Cloudflare</title></title>
...

It looks like our kaniko build step that was creating our image got a 503 when fetching one of the layers from docker.com. I'm not sure if there's anything we can do to fix this. I think not.

But I'm filing this issue anyway so we can track if it's a common issue.

The text was updated successfully, but these errors were encountered:

coryan · 2021-05-03T15:40:35Z

Seems like this could be useful:

https://github.com/GoogleContainerTools/kaniko/blob/master/README.md#--registry-mirror

coryan · 2021-05-04T12:46:30Z

I tried adding --registry-mirror=mirror.gcr.io and did not work. From what I could gather from the error messages, mirror.gcr.io does not host a number of images we need (Fedora:33, or Ubuntu:bionic).

There is a way to create our own mirror and host it, but that seems very involved.

coryan · 2021-05-08T01:17:56Z

This one is similar enough that I think we should consolidate them:

https://pantheon.corp.google.com/cloud-build/builds;region=global/9ef522b9-59b8-4ca9-9a71-0d5789a497ee;step=0?project=cloud-cpp-testing-resources

error building image: GET https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/ae/aefd7f02ae24739b95f77c488de70465c54653f394097b9859acede976c80e03/data?verify=REDACTED: unsupported status code 502; body: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>cloudflare</center>
</body>
</html>

coryan · 2021-05-10T14:23:19Z

I figured out why --registry-mirror=mirror.gcr.io does not work: it just holds "latest" of each popular image, bummer.

coryan · 2021-05-10T14:33:18Z

A different solution may involve using the "warmer" program:

https://github.com/GoogleContainerTools/kaniko/tree/master/cmd/warmer

This can download the base image to /cache which could be a shared volume between the warmer and the kaniko steps.

There are a couple of additional twists:

Saying something like fedora:33 or ubuntu:bionic requires a roundtrip to registry.github.io because those versions may (and do) change.
These roundtrips to registry.github.io are what fails with 5xx errors
We can avoid them (to some degree) if we pin a SHA, and say FROM fedora:33@sha256:ab9c680acef5a053cf2a6bddcebfa9674576d5104927180ef27a35d2dbab15fc
Note that using the SHA saves a roundtrip to the registry even if we do not use the warmer, in other words, one less chance to get snake eyes when rolling the network dice.
Both the warmer and the kaniko steps would need to download the same SHA, the warmer takes a docker image name, not a dockerfile as input, that suggests a script to extract the version from the Dockerfile
We can have the renovate bot update the SHA, that in itself seems interesting, because maybe we do not want updates to the OS unless we run the tests first.
Note that the warmer step would still need to download the base image, so we are saving roundtrips to the registry, but still have a download.
Maybe we can have that /cache directory really cached as a GCS tarball (sure would be nice if kaniko did that instead).

coryan · 2021-09-23T13:04:55Z

No repeats in 90d, closing. I suspect we will need to reopen though.

devbww · 2022-02-19T22:19:02Z

https://console.cloud.google.com/cloud-build/builds;region=us-east1/99910c10-0808-429c-9f10-4262c19416ba?project=cloud-cpp-testing-resources

Step #0: error building image: GET https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/dc/dcf4d4bef137f695d11ed187ba6a135362dca3de36955c4da0905d596ce521bc/data?verify=REDACTED: unexpected status code 502 Bad Gateway: <html>
Step #0: <head><title>502 Bad Gateway</title></head>
Step #0: <body>
Step #0: <center><h1>502 Bad Gateway</h1></center>
Step #0: <hr><center>cloudflare</center>
Step #0: </body>
Step #0: </html>

devjgm · 2022-03-16T14:42:41Z

https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/472c831b-dd84-4d4d-98a5-7274a880d8b3;step=0?project=cloud-cpp-testing-resources

error building image: error building stage: failed to get filesystem from image: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/7c/7c3b88808835aa80f1ef7f03083c5ae781d0f44e644537cd72de4ce6c5e62e00/data?verify=1647437859-y6QANhynFYEw1DOP0fkes9J%2F4eY%3D": read tcp 192.168.10.3:54192->104.18.124.25:443: read: connection reset by peer

coryan · 2022-03-16T14:52:35Z

Maybe this can help:

https://github.com/GoogleContainerTools/kaniko/blob/v1.8.0/README.md#--image-fs-extract-retry

Related to googleapis#6438 This might (or might not) fix the rare kaniko flakes we see. But I imagine it almost certainly cannot hurt.

devjgm · 2022-03-16T21:18:05Z

Nice find. Can't hurt. Let's give it a shot: #8558

Related to #6438 This might (or might not) fix the rare kaniko flakes we see. But I imagine it almost certainly cannot hurt.

Related to googleapis#6438 This might (or might not) fix the rare kaniko flakes we see. But I imagine it almost certainly cannot hurt.

dbolduc · 2022-05-07T12:33:19Z

This was a 404
https://console.cloud.google.com/cloud-build/builds;region=us-east1/a24f0fba-ce12-43bd-923f-d0a1964f1442?project=936212892354

coryan · 2022-08-06T16:01:01Z

90d without a repeat, closing.

dbolduc · 2023-06-12T19:26:18Z

CONSOLE_LOG_URL
https://console.cloud.google.com/cloud-build/builds;region=us-east1/bdf61932-1f3a-48f0-9a11-aec2059c0b25;tab=detail?project=cloud-cpp-testing-resources
RAW_LOG_URL
https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/c33459e640286d3ff0c7a51c1f66bc6e0d2acb36/demo-debian-bullseye-demo-install/log-bdf61932-1f3a-48f0-9a11-aec2059c0b25.txt

coryan · 2023-09-06T15:03:49Z

Slightly different error message, but I think the same root cause. I am changing the title to be more generic.

https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/542ff150-ac6a-4420-8692-b084b5c5e189?project=cloud-cpp-testing-resources&mods=logs_tg_prod

Step #0: error building image: error building stage: failed to execute command: extracting fs from image: read tcp 192.168.10.2:52280->142.251.162.207:443: read: connection reset by peer

dbolduc · 2023-09-27T23:59:33Z

https://storage.cloud.google.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/12747/73b2c7b43aa5faa972bddffdf524ff72ed81f9fc/fedora-latest-cxx14-cxx14/log-1f83671f-909d-470d-9591-5e4a7d585647.txt

coryan · 2023-10-12T12:33:08Z

https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/94fd1e7a-7f1a-4952-bec3-29a5cd2466a2;step=0?e=-13802955&mods=logs_tg_prod&project=cloud-cpp-testing-resources

devbww · 2023-10-17T02:02:09Z

https://console.cloud.google.com/cloud-build/builds;region=us-east1/9b967be5-c8a8-4871-9c83-407660b496dc?project=cloud-cpp-testing-resources

devbww · 2023-10-20T00:26:40Z

https://console.cloud.google.com/cloud-build/builds;region=us-east1/226f040c-17ef-4b65-b68d-d410e818372a?project=936212892354

error building image: error building stage: failed to execute command: extracting fs from image: read tcp 192.168.10.2:57314->74.125.26.207:443: read: connection reset by peer

coryan · 2023-10-20T01:04:34Z

FWIW: this seems to be:

GoogleContainerTools/kaniko#1717

This may also be of use, but requires a lot more configuration:

https://cloud.google.com/artifact-registry/docs/repositories/remote-repo

dbolduc · 2023-10-30T21:59:04Z

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/7cb1ec3ee393fefe431bea87a872063c78f722cc/fedora-msan-msan/log-2e3dd95c-73bc-4a4c-a66c-6ee3a20c9201.txt

dbolduc · 2023-11-02T17:43:40Z

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/0cf8a7a77daca027399b1f590c1c1cd02ef2a4cb/fedora-latest-cmake-check-api/log-fd678bdb-61ce-4bef-9605-75e5db60582e.txt

error building image: error building stage: failed to execute command: extracting fs from image: unexpected EOF

alevenberg · 2023-11-09T17:08:59Z

Attempting to fix upstream (GoogleContainerTools/kaniko#2837)

coryan · 2023-11-17T13:00:11Z

https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/163e2878-c35f-4dfb-bfae-7770b55dc1d8;step=0?e=-13802955&mods=logs_tg_prod&project=cloud-cpp-testing-resources

dbolduc · 2023-12-12T03:15:52Z

error building image: error building stage: failed to execute command: extracting fs from image: read tcp 192.168.10.2:60614->108.177.12.207:443: read: connection reset by peer

alevenberg · 2023-12-13T15:00:09Z

I tried :(

devbww · 2023-12-26T19:55:47Z

https://console.cloud.google.com/cloud-build/builds;region=us-east1/81179ad8-943b-4d47-9f5b-e5ad5c7ebeea?project=cloud-cpp-testing-resources&pli=1&e=-13802955&mods=dataproc_env_prod

dbolduc · 2024-01-31T19:06:14Z

CONSOLE_LOG_URL
https://console.cloud.google.com/cloud-build/builds;region=us-east1/8f5cd35a-5c37-4d18-8f82-97283c00d633;tab=detail?project=cloud-cpp-testing-resources
RAW_LOG_URL
https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/94355b0efb9e087c241c2be3cca062e818d5a7dd/fedora-latest-bazel-tsan/log-8f5cd35a-5c37-4d18-8f82-97283c00d633.txt

alevenberg · 2024-02-13T16:09:09Z

Step #0: error building image: error building stage: failed to execute command: extracting fs from image: read tcp 192.168.10.2:45946->74.125.196.207:443: read: connection reset by peer

Build FAILURE: libcxx-ci https://console.cloud.google.com/cloud-build/builds;region=us-east1/aea5600b-a3a6-4159-a043-3ae3d52d8dac?project=936212892354

https://ff61d7d2b82917f3c17eaeff0c86b71d3239f0c72a9c7b4acb459bf-apidata.googleusercontent.com/download/storage/v1/b/cloud-cpp-community-publiclogs/o/logs%2Fgoogle-cloud-cpp%2Fmain%2Fa988f3e39ca5134b6966578c3db5da07e1147156%2Ffedora-latest-bazel-libcxx-__default__%2Flog-aea5600b-a3a6-4159-a043-3ae3d52d8dac.txt?jk=ATxpoHeeaWtiteuzk_Hv4q9-IM7XMf4UI-FwcW3nzSl-qCF_T09zhJSr_WlG3yIzztYXmDjGnzEOuwJO3645otW7Tk0IFqhjohyC6X70Ww9uuVpgOE_C9AiMvxoM-12lPeUJmAWVgzFYLUJ_sbEGjmRnksixk2RozK04efDNH5GQiHCvwkfpXwj61Ex_YIuqmFt5ji7gB0HbVfDMzcBD8DZZpWzhoAPZzytpxVDoJj1UcwOvTUaApiNrREyaAzHnH5R5ilGLZn6x6Be4kW_LqLMQWxDOMcWVAFVntzW3E9WRjA3XuxNB8B7lUDvBC1POWwCH52Lfs_DhnrKqTL-O9GBQvKaicZ_ubfd7U2SSK3UOQIf4pxmBmSVbTiSUqCQebhTAwqBZio5MM6wjiytRtr9GkPu3Ld52JZvOPHcnwPkA7BFzw351pZfgh9SrQo2qAv2z58X8OdZua32RPvTBb3jXN3MZJ4ZL55eACl6ctTbNn3aXV0DiB6oiMKc7fvBGdX3RQhH6Qex0vFiCHEY9fq7xan_q1TVcRGuxRnYK322NsL33vqY2OIVxmpZBOpKvnMiZRMpaFp3zGlry-S-waYcXTaxIk_cfj8Hv857mA9aeG2GjMigIZxsXlBs6wGiuIxoY8Tg0r8cjN0QJ7q4Ub0K2B3m-vZM7dY3-6lAbk8lC9cpiAkKaiE9neOf8wOYVSPU1-gcXBqz6W7G2pa8nEuG5HTtSMOsWDhLGo3Prv0U3o7VMtkVyomZtf4ubtRjU3WPGwMEmDk9TdQJGkPKzs7UFqwzJwNeW5dbnU8EzgTN9s9S0TuxxqJaB2xg6_Q-73X8qZdXL7QwgiqI6OK7fDEgC2vD92tj47K_G-Ar5hsbFC3FSPHOXc7PQI1z92WjsuIWlzgaObFlAAslcLtk6q7Kpg36RHYJPEtRFFQcEQuC3PermC-EUQtsL5wOclZXTkh9VHc05qhcnI753DF5FOAbFOwytdoz2K_wYHyNc4TWUI9YZzuAhsDUseayDvlZEN_fm4n88kgFz-w2aGzPvH61SKblV_qttdslHvIHi881RKdFV7nkNghHjQaF97CPcipl6DhMqm6sW7Yl8bhguSIuktyAQ2xZosH-QpKo5HhGS8spa-k6FNvw7GOvG7EjEDYJvAKJL0PN4HQtKzKmTxjMYyq4OKcR1LgqpSpeREY_ne7vHBnhXIn1OBXOXoZ3cPuk9pR5M5ZAFvIfvzmmGDeTsnPx8V6nam8I1oBxyeeblL7PeTZzXLyFVx3STt0l3XCFvvU32eEkWahxrygI_FdAmg4URqy06igc88ofMTqjZN6MPDoF3IYNNzYOSEGdUvF2kGCV60r7Hm-zQ4V6jg3hGy-7gLUeNKsmzraqqAaIQ2iBoN9Rv7av6&isca=1

coryan · 2024-02-20T19:15:12Z

https://console.cloud.google.com/cloud-build/builds;region=us-east1/93b44eb8-5bd9-4ca1-910f-6e8d5d0e09d7?project=936212892354

coryan · 2024-02-20T22:16:36Z

https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/f20dda71-8239-469b-b434-01a37df4349d?project=cloud-cpp-testing-resources&e=VertexAiColabPublicPreviewLaunch::VertexAiColabPublicPreviewEnabled&mods=logs_tg_prod

coryan · 2024-02-23T14:11:37Z

https://console.cloud.google.com/cloud-build/builds;region=us-east1/c2a442b3-f2b4-4eb2-8137-f0bc5125ecfc;tab=detail?project=cloud-cpp-testing-resources

devbww · 2024-02-27T16:19:42Z

https://console.cloud.google.com/cloud-build/builds;region=us-east1/d443c4c9-09ce-4b08-92e1-a024a32c5f8a?project=cloud-cpp-testing-resources

devbww · 2024-02-28T02:15:43Z

https://console.cloud.google.com/cloud-build/builds;region=us-east1/713414dc-9704-4327-b6a6-d647552c7b61?project=936212892354

devbww · 2024-02-29T18:38:17Z

https://console.cloud.google.com/cloud-build/builds;region=us-east1/50d675f5-30b6-494c-9512-fd8bf0029a97?project=936212892354

scotthart · 2024-09-11T19:38:00Z

Haven't seen in 90d; closing.

devjgm added the type: cleanup An internal cleanup or hygiene concern. label May 3, 2021

coryan changed the title ~~[Flake] kaniko flake (503) fetching image from production.cloudflare.docker.com~~ [Flake] kaniko flake (5xx) fetching image from production.cloudflare.docker.com May 8, 2021

coryan closed this as completed Sep 23, 2021

coryan added the cpp: flake A test with false positives (failures that are not interesting) label Jan 27, 2022

devbww reopened this Feb 19, 2022

devjgm added a commit to devjgm/google-cloud-cpp that referenced this issue Mar 16, 2022

ci: enable retries for kaniko

da67d2b

Related to googleapis#6438 This might (or might not) fix the rare kaniko flakes we see. But I imagine it almost certainly cannot hurt.

devjgm mentioned this issue Mar 16, 2022

ci: enable retries for kaniko #8558

Merged

devjgm added a commit that referenced this issue Mar 17, 2022

ci: enable retries for kaniko (#8558)

64c75a2

Related to #6438 This might (or might not) fix the rare kaniko flakes we see. But I imagine it almost certainly cannot hurt.

devjgm added a commit to devjgm/google-cloud-cpp that referenced this issue Mar 17, 2022

ci: enable retries for kaniko

d10dd2c

Related to googleapis#6438 This might (or might not) fix the rare kaniko flakes we see. But I imagine it almost certainly cannot hurt.

coryan closed this as completed Aug 6, 2022

dbolduc reopened this Jun 12, 2023

coryan mentioned this issue Oct 26, 2023

impl(pubsub): use semantic convention for attribute #12977

Merged

alevenberg self-assigned this Nov 9, 2023

coryan mentioned this issue Nov 13, 2023

Build stops with Get "https://index.docker.io/v2/": EOF #13113

Closed

alevenberg mentioned this issue Nov 20, 2023

fix(ci): add a retry loop on the image download #13165

Merged

alevenberg closed this as completed in #13165 Nov 30, 2023

dbolduc reopened this Dec 12, 2023

alevenberg removed their assignment Dec 13, 2023

This was referenced Feb 20, 2024

ci(gcb): make 2 kaniko attempts #13639

Merged

ci(gcb): the first kaniko attempt can fail #13652

Merged

coryan mentioned this issue May 10, 2024

ci(gcb): make multiple Kaniko attempts #14222

Merged

scotthart closed this as completed Sep 11, 2024

[Flake] kaniko flake (5xx) fetching image from production.cloudflare.docker.com #6438

[Flake] kaniko flake (5xx) fetching image from production.cloudflare.docker.com #6438

Comments

devjgm commented May 3, 2021

coryan commented May 3, 2021

coryan commented May 4, 2021

coryan commented May 8, 2021

coryan commented May 10, 2021

coryan commented May 10, 2021

coryan commented Sep 23, 2021

devbww commented Feb 19, 2022

devjgm commented Mar 16, 2022 • edited Loading

coryan commented Mar 16, 2022

devjgm commented Mar 16, 2022

dbolduc commented May 7, 2022 • edited Loading

coryan commented Aug 6, 2022

dbolduc commented Jun 12, 2023

coryan commented Sep 6, 2023

dbolduc commented Sep 27, 2023

coryan commented Oct 12, 2023

devbww commented Oct 17, 2023

devbww commented Oct 20, 2023

coryan commented Oct 20, 2023

dbolduc commented Oct 30, 2023

dbolduc commented Nov 2, 2023

alevenberg commented Nov 9, 2023

coryan commented Nov 17, 2023

dbolduc commented Dec 12, 2023

alevenberg commented Dec 13, 2023

devbww commented Dec 26, 2023

dbolduc commented Jan 31, 2024

alevenberg commented Feb 13, 2024 • edited Loading

coryan commented Feb 20, 2024

coryan commented Feb 20, 2024

coryan commented Feb 23, 2024

devbww commented Feb 27, 2024

devbww commented Feb 28, 2024

devbww commented Feb 29, 2024

scotthart commented Sep 11, 2024

devjgm commented Mar 16, 2022 •

edited

Loading

dbolduc commented May 7, 2022 •

edited

Loading

alevenberg commented Feb 13, 2024 •

edited

Loading