Cannot push image to Internal Registry

[mibaboo]

Hi

I have followed this guide to push an image into the OpenShift internal registry. I am able to login to the registry however I cannot push.

docker push docker-registry-default.10.28.102.29.nip.io/pushed/myimage:latest

The push refers to a repository [docker-registry-default.10.28.102.29.nip.io/pushed/myimage]
f999ae22f308: Retrying in 1 second
Error: Status 503 trying to push repository pushed/myimage: "\n \n <meta name="viewport" content="width=device-width, initial-scale=1">\n\n <style type="text/css">\n /*!\n * Bootstrap v3.3.5 (http://getbootstrap.com)\n * Copyright 2011-2015 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n /\n /! normalize.css v3.0.3 | MIT License | github.com/necolas/normalize.css */\n html {\n font-family: sans-serif;\n -ms-text-size-adjust: 100%;\n -webkit-text-size-adjust: 100%;\n }\n body {\n margin: 0;\n }\n h1 {\n font-size: 1.7em;\n font-weight: 400;\n line-height: 1.3;\n margin: 0.68em 0;\n }\n * {\n -webkit-box-sizing: border-box;\n -moz-box-sizing: border-box;\n box-sizing: border-box;\n }\n *:before,\n *:after {\n -webkit-box-sizing: border-box;\n -moz-box-sizing: border-box;\n box-sizing: border-box;\n }\n html {\n -webkit-tap-highlight-color: rgba(0, 0, 0, 0);\n }\n body {\n font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\n line-height: 1.66666667;\n font-size: 13px;\n color: #333333;\n background-color: #ffffff;\n margin: 2em 1em;\n }\n p {\n margin: 0 0 10px;\n font-size: 13px;\n }\n .alert.alert-info {\n padding: 15px;\n margin-bottom: 20px;\n border: 1px solid transparent;\n background-color: #f5f5f5;\n border-color: #8b8d8f;\n color: #363636;\n margin-top: 30px;\n }\n .alert p {\n padding-left: 35px;\n }\n a {\n color: #0088ce;\n }\n\n ul {\n position: relative;\n padding-left: 51px;\n }\n p.info {\n position: relative;\n font-size: 15px;\n margin-bottom: 10px;\n }\n p.info:before, p.info:after {\n content: "";\n position: absolute;\n top: 9%;\n left: 0;\n }\n p.info:before {\n content: "i";\n left: 3px;\n width: 20px;\n height: 20px;\n font-family: serif;\n font-size: 15px;\n font-weight: bold;\n line-height: 21px;\n text-align: center;\n color: #fff;\n background: #4d5258;\n border-radius: 16px;\n }\n\n @media (min-width: 768px) {\n body {\n margin: 4em 3em;\n }\n h1 {\n font-size: 2.15em;}\n }\n\n </style>\n \n \n

\n

Application is not available

\n

The application is currently not serving requests at this endpoint. It may not have been started or is still starting.

\n\n <div class="alert alert-info">\n <p class="info">\n Possible reasons you are seeing this page:\n

\n

\n
• \n The host doesn't exist.\n Make sure the hostname was typed correctly and that a route matching this hostname exists.\n
\n
• \n The host exists, but doesn't have a matching path.\n Check if the URL path was typed correctly and that the route was created using the desired path.\n
\n
• \n Route and path matches, but all pods are down.\n Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\n
\n

\n

\n \n \n\n"

Version

oc version
oc v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://10.28.102.29:8443
openshift v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7

openshift version
openshift v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

Steps To Reproduce

oc cluster up --public-hostname 10.28.102.29 --host-data-dir=/opt/openshift/data/ --host-config-dir=/opt/openshift/config/ --use-existing-config --http-proxy=http://USER:PASS@PROXY:8080 --https-proxy=http://USER:PASS@PROXY:8080 --no-proxy=172.30.1.1
oc create serviceaccount pusher
oc policy add-role-to-user system:image-builder pusher

oc create -f - <<API

apiVersion: v1

kind: ImageStream

metadata:

  annotations:

    description: Keeps track of changes in the application image

  name: myimage

API

docker login -u pusher -p @&@&@ docker-registry-default.10.28.102.29.nip.io

docker tag f2a91732366c docker-registry-default.10.28.102.29.nip.io/pushed/myimage:latest

docker push docker-registry-default.172.28.102.29.nip.io/pushed/myimage:latest

Current Result

The push refers to a repository [docker-registry-default.10.28.102.29.nip.io/pushed/myimage]
f999ae22f308: Retrying in 1 second
Error: Status 503 trying to push repository pushed/myimage: "\n \n <meta name="viewport" content="width=device-width, initial-scale=1">\n\n <style type="text/css">\n /*!\n * Bootstrap v3.3.5 (http://getbootstrap.com)\n * Copyright 2011-2015 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n /\n /! normalize.css v3.0.3 | MIT License | github.com/necolas/normalize.css */\n html {\n font-family: sans-serif;\n -ms-text-size-adjust: 100%;\n -webkit-text-size-adjust: 100%;\n }\n body {\n margin: 0;\n }\n h1 {\n font-size: 1.7em;\n font-weight: 400;\n line-height: 1.3;\n margin: 0.68em 0;\n }\n * {\n -webkit-box-sizing: border-box;\n -moz-box-sizing: border-box;\n box-sizing: border-box;\n }\n *:before,\n *:after {\n -webkit-box-sizing: border-box;\n -moz-box-sizing: border-box;\n box-sizing: border-box;\n }\n html {\n -webkit-tap-highlight-color: rgba(0, 0, 0, 0);\n }\n body {\n font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\n line-height: 1.66666667;\n font-size: 13px;\n color: #333333;\n background-color: #ffffff;\n margin: 2em 1em;\n }\n p {\n margin: 0 0 10px;\n font-size: 13px;\n }\n .alert.alert-info {\n padding: 15px;\n margin-bottom: 20px;\n border: 1px solid transparent;\n background-color: #f5f5f5;\n border-color: #8b8d8f;\n color: #363636;\n margin-top: 30px;\n }\n .alert p {\n padding-left: 35px;\n }\n a {\n color: #0088ce;\n }\n\n ul {\n position: relative;\n padding-left: 51px;\n }\n p.info {\n position: relative;\n font-size: 15px;\n margin-bottom: 10px;\n }\n p.info:before, p.info:after {\n content: "";\n position: absolute;\n top: 9%;\n left: 0;\n }\n p.info:before {\n content: "i";\n left: 3px;\n width: 20px;\n height: 20px;\n font-family: serif;\n font-size: 15px;\n font-weight: bold;\n line-height: 21px;\n text-align: center;\n color: #fff;\n background: #4d5258;\n border-radius: 16px;\n }\n\n @media (min-width: 768px) {\n body {\n margin: 4em 3em;\n }\n h1 {\n font-size: 2.15em;}\n }\n\n </style>\n \n \n

\n

Application is not available

\n

The application is currently not serving requests at this endpoint. It may not have been started or is still starting.

\n\n <div class="alert alert-info">\n <p class="info">\n Possible reasons you are seeing this page:\n

\n

\n
• \n The host doesn't exist.\n Make sure the hostname was typed correctly and that a route matching this hostname exists.\n
\n
• \n The host exists, but doesn't have a matching path.\n Check if the URL path was typed correctly and that the route was created using the desired path.\n
\n
• \n Route and path matches, but all pods are down.\n Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\n
\n

\n

\n \n \n\n"

Expected Result

images pushed successfully

Additional Information

oc status
In project default on server https://10.28.102.29:8443

http://docker-registry-default.10.28.102.29.nip.io to pod port 5000-tcp (svc/docker-registry)
dc/docker-registry deploys docker.io/openshift/origin-docker-registry:v3.6.1
deployment #1 deployed 4 hours ago - 1 pod

svc/kubernetes - 10.30.0.1 ports 443->8443, 53->8053, 53->8053

svc/router - 10.30.177.62 ports 80, 443, 1936
dc/router deploys docker.io/openshift/origin-haproxy-router:v3.6.1
deployment #1 deployed 4 hours ago - 1 pod

View details with 'oc describe /' or list everything with 'oc get all'.

[if you are reporting issue related to builds, provide build logs with BUILD_LOGLEVEL=5]
[consider attaching output of the $ oc get all -o json -n <namespace> command to the issue]
[visit https://docs.openshift.org/latest/welcome/index.html]

[mfojtik]

/assign bparees
/priority P1

[bparees]

@mfojtik why p1?

[bparees]

Seems like your registry pod is not running. Please confirm it is running and gather the logs from it.

[mibaboo]

The registry pod is running
docker-registry-1-2tp44 1/1 Running 0 1d

logs

172.17.0.1 - - [06/Dec/2017:08:30:26 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:30:26 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:30:36 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:30:36 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:30:46 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:30:46 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:30:56 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:30:56 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:31:06 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:31:06 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:31:16 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:31:16 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:31:26 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:31:26 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:31:36 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:31:36 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:31:46 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1" 172.17.0.1 - - [06/Dec/2017:08:31:46 +0000] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1"

[albgus]

I had the same issue. Turned out that it was because my load balancer (AWS ALB) in front of the routers doesn't pass TLS SNI headers. Since the registry and console is using a passthrough route the routers only look at the SNI header.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[alexandrev]

I'm facing same issue right now, as well as #12863 . It seems this bug is closed over and over without providing a real solution.

[bparees]

I'm facing same issue right now, as well as #12863 . It seems this bug is closed over and over without providing a real solution.

because every person who hits this is hitting it due to unique configuration issues in their environment, there is no generalized problem w/ pushing images to the internal registry (it's a fundamental feature of openshift, it is well tested and works consistently), and if the person hitting the issue isn't responsive to our queries for more information to help resolve it, there is nothing else we can do to help them.

Please open your own issue describing your cluster configuration, error you are hitting, and providing registry logs, if you would like assistance.

[lrhazi]

The comment above about TLS SNI was key for me... I had same issue, https traffic was being terminated and then re-encrypted by a loadbalancer... I made the LB pass traffic through, without terminating ssl, and the issue is gone! Am sure this documented somewhere? that kube needs TLS SNI to identify the hostnames and be able to route traffic. Thank you!

[bparees]

@lrhazi https://docs.openshift.org/latest/dev_guide/expose_service/expose_internal_ip_router.html#overview

"A router is configured to accept external requests and proxy them based on the configured routes. This is limited to HTTP/HTTPS(SNI)/TLS(SNI), which covers web applications."

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close