[porter/kubeflow] Kubeflow installation fails

[MikeEvansLarah]

Expected Behavior

Kubeflow installation succeeds

Actual Behavior

Kubeflow installation gets stuck with following error:

time="2020-04-07T14:32:30Z" level=warning msg="Encountered error applying application cert-manager:  (kubeflow.error): C
ode 500 with message: Apply.Run  Error [error when applying patch:\n{\"metadata\":{\"annotations\":{\"kubectl.kubernetes
.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"apps/v1\\\",\\\"kind\\\":\\\"Deployment\\\",\\\"metadata\\\":
{\\\"annotations\\\":{},\\\"labels\\\":{\\\"app\\\":\\\"cert-manager\\\",\\\"app.kubernetes.io/component\\\":\\\"cert-ma
nager\\\",\\\"app.kubernetes.io/instance\\\":\\\"cert-manager-v1.0.0\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"kfctl\
\\",\\\"app.kubernetes.io/name\\\":\\\"cert-manager\\\",\\\"app.kubernetes.io/part-of\\\":\\\"kubeflow\\\",\\\"app.kuber
netes.io/version\\\":\\\"v1.0.0\\\",\\\"kustomize.component\\\":\\\"cert-manager\\\"},\\\"name\\\":\\\"cert-manager\\\",
\\\"namespace\\\":\\\"cert-manager\\\"},\\\"spec\\\":{\\\"replicas\\\":1,\\\"selector\\\":{\\\"matchLabels\\\":{\\\"app\
\\":\\\"cert-manager\\\",\\\"app.kubernetes.io/component\\\":\\\"cert-manager\\\",\\\"app.kubernetes.io/instance\\\":\\\
"cert-manager-v1.0.0\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"kfctl\\\",\\\"app.kubernetes.io/name\\\":\\\"cert-mana
ger\\\",\\\"app.kubernetes.io/part-of\\\":\\\"kubeflow\\\",\\\"app.kubernetes.io/version\\\":\\\"v1.0.0\\\",\\\"kustomiz
e.component\\\":\\\"cert-manager\\\"}},\\\"template\\\":{\\\"metadata\\\":{\\\"annotations\\\":{\\\"prometheus.io/path\\
\":\\\"/metrics\\\",\\\"prometheus.io/port\\\":\\\"9402\\\",\\\"prometheus.io/scrape\\\":\\\"true\\\"},\\\"labels\\\":{\
\\"app\\\":\\\"cert-manager\\\",\\\"app.kubernetes.io/component\\\":\\\"cert-manager\\\",\\\"app.kubernetes.io/instance\
\\":\\\"cert-manager-v1.0.0\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"kfctl\\\",\\\"app.kubernetes.io/name\\\":\\\"ce
rt-manager\\\",\\\"app.kubernetes.io/part-of\\\":\\\"kubeflow\\\",\\\"app.kubernetes.io/version\\\":\\\"v1.0.0\\\",\\\"k
ustomize.component\\\":\\\"cert-manager\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"args\\\":[\\\"--v=2\\\",\\\"--clust
er-resource-namespace=$(POD_NAMESPACE)\\\",\\\"--leader-election-namespace=kube-system\\\",\\\"--webhook-namespace=$(POD
_NAMESPACE)\\\",\\\"--webhook-ca-secret=cert-manager-webhook-ca\\\",\\\"--webhook-serving-secret=cert-manager-webhook-tl
s\\\",\\\"--webhook-dns-names=cert-manager-webhook,cert-manager-webhook.cert-manager,cert-manager-webhook.cert-manager.s
vc\\\"],\\\"env\\\":[{\\\"name\\\":\\\"POD_NAMESPACE\\\",\\\"valueFrom\\\":{\\\"fieldRef\\\":{\\\"fieldPath\\\":\\\"meta
data.namespace\\\"}}}],\\\"image\\\":\\\"quay.io/jetstack/cert-manager-controller:v0.11.0\\\",\\\"imagePullPolicy\\\":\\
\"IfNotPresent\\\",\\\"name\\\":\\\"cert-manager\\\",\\\"ports\\\":[{\\\"containerPort\\\":9402}],\\\"resources\\\":{\\\
"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"32Mi\\\"}}}],\\\"serviceAccountName\\\":\\\"cert-manager\\\"}}
}}\\n\"},\"labels\":{\"app.kubernetes.io/component\":\"cert-manager\",\"app.kubernetes.io/instance\":\"cert-manager-v1.0
.0\",\"app.kubernetes.io/managed-by\":\"kfctl\",\"app.kubernetes.io/name\":\"cert-manager\",\"app.kubernetes.io/part-of\
":\"kubeflow\",\"app.kubernetes.io/version\":\"v1.0.0\",\"kustomize.component\":\"cert-manager\"}},\"spec\":{\"selector\
":{\"matchLabels\":{\"app.kubernetes.io/component\":\"cert-manager\",\"app.kubernetes.io/instance\":\"cert-manager-v1.0.
0\",\"app.kubernetes.io/managed-by\":\"kfctl\",\"app.kubernetes.io/name\":\"cert-manager\",\"app.kubernetes.io/part-of\"
:\"kubeflow\",\"app.kubernetes.io/version\":\"v1.0.0\",\"kustomize.component\":\"cert-manager\"}},\"template\":{\"metada
ta\":{\"labels\":{\"app.kubernetes.io/component\":\"cert-manager\",\"app.kubernetes.io/instance\":\"cert-manager-v1.0.0\
",\"app.kubernetes.io/managed-by\":\"kfctl\",\"app.kubernetes.io/name\":\"cert-manager\",\"app.kubernetes.io/part-of\":\
"kubeflow\",\"app.kubernetes.io/version\":\"v1.0.0\",\"kustomize.component\":\"cert-manager\"}},\"spec\":{\"$setElementO
rder/containers\":[{\"name\":\"cert-manager\"}],\"containers\":[{\"args\":[\"--v=2\",\"--cluster-resource-namespace=$(PO
D_NAMESPACE)\",\"--leader-election-namespace=kube-system\",\"--webhook-namespace=$(POD_NAMESPACE)\",\"--webhook-ca-secre
t=cert-manager-webhook-ca\",\"--webhook-serving-secret=cert-manager-webhook-tls\",\"--webhook-dns-names=cert-manager-web
hook,cert-manager-webhook.cert-manager,cert-manager-webhook.cert-manager.svc\"],\"image\":\"quay.io/jetstack/cert-manage
r-controller:v0.11.0\",\"name\":\"cert-manager\",\"resources\":{\"requests\":{\"cpu\":\"10m\",\"memory\":\"32Mi\"}}}]}}}
}\nto:\nResource: \"apps/v1, Resource=deployments\", GroupVersionKind: \"apps/v1, Kind=Deployment\"\nName: \"cert-manage
r\", Namespace: \"cert-manager\"\nObject: &{map[\"apiVersion\":\"apps/v1\" \"kind\":\"Deployment\" \"metadata\":map[\"an
notations\":map[\"deployment.kubernetes.io/revision\":\"1\"] \"creationTimestamp\":\"2020-01-21T10:03:48Z\" \"generation
\":'\\x01' \"labels\":map[\"app\":\"cert-manager\" \"chart\":\"cert-manager-v0.8.0\" \"heritage\":\"Tiller\" \"release\"
:\"cert-manager\"] \"name\":\"cert-manager\" \"namespace\":\"cert-manager\" \"resourceVersion\":\"21349820\" \"selfLink\
":\"/apis/apps/v1/namespaces/cert-manager/deployments/cert-manager\" \"uid\":\"5169f7cd-3c35-11ea-baff-eee59f452b94\"] \
"spec\":map[\"progressDeadlineSeconds\":'\\u0258' \"replicas\":'\\x01' \"revisionHistoryLimit\":'\\x02' \"selector\":map
[\"matchLabels\":map[\"app\":\"cert-manager\" \"release\":\"cert-manager\"]] \"strategy\":map[\"rollingUpdate\":map[\"ma
xSurge\":\"25%\" \"maxUnavailable\":\"25%\"] \"type\":\"RollingUpdate\"] \"template\":map[\"metadata\":map[\"annotations
\":map[\"prometheus.io/path\":\"/metrics\" \"prometheus.io/port\":\"9402\" \"prometheus.io/scrape\":\"true\"] \"creation
Timestamp\":<nil> \"labels\":map[\"app\":\"cert-manager\" \"release\":\"cert-manager\"]] \"spec\":map[\"containers\":[ma
p[\"args\":[\"--v=2\" \"--cluster-resource-namespace=$(POD_NAMESPACE)\" \"--leader-election-namespace=$(POD_NAMESPACE)\"
] \"env\":[map[\"name\":\"POD_NAMESPACE\" \"valueFrom\":map[\"fieldRef\":map[\"apiVersion\":\"v1\" \"fieldPath\":\"metad
ata.namespace\"]]]] \"image\":\"quay.io/jetstack/cert-manager-controller:v0.8.0\" \"imagePullPolicy\":\"IfNotPresent\" \
"name\":\"cert-manager\" \"ports\":[map[\"containerPort\":'\\u24ba' \"protocol\":\"TCP\"]] \"resources\":map[] \"termina
tionMessagePath\":\"/dev/termination-log\" \"terminationMessagePolicy\":\"File\"]] \"dnsPolicy\":\"ClusterFirst\" \"rest
artPolicy\":\"Always\" \"schedulerName\":\"default-scheduler\" \"securityContext\":map[] \"serviceAccount\":\"cert-manag
er\" \"serviceAccountName\":\"cert-manager\" \"terminationGracePeriodSeconds\":'\\x1e']]] \"status\":map[\"availableRepl
icas\":'\\x01' \"conditions\":[map[\"lastTransitionTime\":\"2020-01-21T10:03:48Z\" \"lastUpdateTime\":\"2020-01-21T10:03
:51Z\" \"message\":\"ReplicaSet \\\"cert-manager-5d669ffbd8\\\" has successfully progressed.\" \"reason\":\"NewReplicaSe
tAvailable\" \"status\":\"True\" \"type\":\"Progressing\"] map[\"lastTransitionTime\":\"2020-04-07T14:32:25Z\" \"lastUpd
ateTime\":\"2020-04-07T14:32:25Z\" \"message\":\"Deployment has minimum availability.\" \"reason\":\"MinimumReplicasAvai
lable\" \"status\":\"True\" \"type\":\"Available\"]] \"observedGeneration\":'\\x01' \"readyReplicas\":'\\x01' \"replicas
\":'\\x01' \"updatedReplicas\":'\\x01']]}\nfor: \"/tmp/kout370059756\": Deployment.apps \"cert-manager\" is invalid: spe
c.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app\":\"cert-manager\", \"app.kubernetes.io/
component\":\"cert-manager\", \"app.kubernetes.io/instance\":\"cert-manager-v1.0.0\", \"app.kubernetes.io/managed-by\":\
"kfctl\", \"app.kubernetes.io/name\":\"cert-manager\", \"app.kubernetes.io/part-of\":\"kubeflow\", \"app.kubernetes.io/v
ersion\":\"v1.0.0\", \"kustomize.component\":\"cert-manager\", \"release\":\"cert-manager\"}, MatchExpressions:[]v1.Labe
lSelectorRequirement(nil)}: field is immutable, error when applying patch:\n{\"metadata\":{\"annotations\":{\"kubectl.ku
bernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"apps/v1\\\",\\\"kind\\\":\\\"Deployment\\\",\\\"metad
ata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"app\\\":\\\"cainjector\\\",\\\"app.kubernetes.io/component\\\":\\\"c
ert-manager\\\",\\\"app.kubernetes.io/instance\\\":\\\"cert-manager-v1.0.0\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"
kfctl\\\",\\\"app.kubernetes.io/name\\\":\\\"cert-manager\\\",\\\"app.kubernetes.io/part-of\\\":\\\"kubeflow\\\",\\\"app
.kubernetes.io/version\\\":\\\"v1.0.0\\\",\\\"kustomize.component\\\":\\\"cert-manager\\\"},\\\"name\\\":\\\"cert-manage
r-cainjector\\\",\\\"namespace\\\":\\\"cert-manager\\\"},\\\"spec\\\":{\\\"replicas\\\":1,\\\"selector\\\":{\\\"matchLab
els\\\":{\\\"app\\\":\\\"cainjector\\\",\\\"app.kubernetes.io/component\\\":\\\"cert-manager\\\",\\\"app.kubernetes.io/i
nstance\\\":\\\"cert-manager-v1.0.0\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"kfctl\\\",\\\"app.kubernetes.io/name\\\
":\\\"cert-manager\\\",\\\"app.kubernetes.io/part-of\\\":\\\"kubeflow\\\",\\\"app.kubernetes.io/version\\\":\\\"v1.0.0\\
\",\\\"kustomize.component\\\":\\\"cert-manager\\\"}},\\\"template\\\":{\\\"metadata\\\":{\\\"annotations\\\":null,\\\"l
abels\\\":{\\\"app\\\":\\\"cainjector\\\",\\\"app.kubernetes.io/component\\\":\\\"cert-manager\\\",\\\"app.kubernetes.io
/instance\\\":\\\"cert-manager-v1.0.0\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"kfctl\\\",\\\"app.kubernetes.io/name\
\\":\\\"cert-manager\\\",\\\"app.kubernetes.io/part-of\\\":\\\"kubeflow\\\",\\\"app.kubernetes.io/version\\\":\\\"v1.0.0
\\\",\\\"kustomize.component\\\":\\\"cert-manager\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"args\\\":[\\\"--v=2\\\",\
\\"--leader-election-namespace=kube-system\\\"],\\\"env\\\":[{\\\"name\\\":\\\"POD_NAMESPACE\\\",\\\"valueFrom\\\":{\\\"
fieldRef\\\":{\\\"fieldPath\\\":\\\"metadata.namespace\\\"}}}],\\\"image\\\":\\\"quay.io/jetstack/cert-manager-cainjecto
r:v0.11.0\\\",\\\"imagePullPolicy\\\":\\\"IfNotPresent\\\",\\\"name\\\":\\\"cainjector\\\",\\\"resources\\\":{}}],\\\"se
rviceAccountName\\\":\\\"cert-manager-cainjector\\\"}}}}\\n\"},\"labels\":{\"app.kubernetes.io/component\":\"cert-manage
r\",\"app.kubernetes.io/instance\":\"cert-manager-v1.0.0\",\"app.kubernetes.io/managed-by\":\"kfctl\",\"app.kubernetes.i
o/name\":\"cert-manager\",\"app.kubernetes.io/part-of\":\"kubeflow\",\"app.kubernetes.io/version\":\"v1.0.0\",\"kustomiz
e.component\":\"cert-manager\"}},\"spec\":{\"selector\":{\"matchLabels\":{\"app.kubernetes.io/component\":\"cert-manager
\",\"app.kubernetes.io/instance\":\"cert-manager-v1.0.0\",\"app.kubernetes.io/managed-by\":\"kfctl\",\"app.kubernetes.io
/name\":\"cert-manager\",\"app.kubernetes.io/part-of\":\"kubeflow\",\"app.kubernetes.io/version\":\"v1.0.0\",\"kustomize
.component\":\"cert-manager\"}},\"template\":{\"metadata\":{\"annotations\":null,\"labels\":{\"app.kubernetes.io/compone
nt\":\"cert-manager\",\"app.kubernetes.io/instance\":\"cert-manager-v1.0.0\",\"app.kubernetes.io/managed-by\":\"kfctl\",
\"app.kubernetes.io/name\":\"cert-manager\",\"app.kubernetes.io/part-of\":\"kubeflow\",\"app.kubernetes.io/version\":\"v
1.0.0\",\"kustomize.component\":\"cert-manager\"}},\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"cainjector\"}]
,\"containers\":[{\"args\":[\"--v=2\",\"--leader-election-namespace=kube-system\"],\"image\":\"quay.io/jetstack/cert-man
ager-cainjector:v0.11.0\",\"name\":\"cainjector\"}]}}}}\nto:\nResource: \"apps/v1, Resource=deployments\", GroupVersionK
ind: \"apps/v1, Kind=Deployment\"\nName: \"cert-manager-cainjector\", Namespace: \"cert-manager\"\nObject: &{map[\"apiVe
rsion\":\"apps/v1\" \"kind\":\"Deployment\" \"metadata\":map[\"annotations\":map[\"deployment.kubernetes.io/revision\":\
"1\"] \"creationTimestamp\":\"2020-01-21T10:03:48Z\" \"generation\":'\\x01' \"labels\":map[\"app\":\"cainjector\" \"char
t\":\"cainjector-v0.8.0\" \"heritage\":\"Tiller\" \"release\":\"cert-manager\"] \"name\":\"cert-manager-cainjector\" \"n
amespace\":\"cert-manager\" \"resourceVersion\":\"21349785\" \"selfLink\":\"/apis/apps/v1/namespaces/cert-manager/deploy
ments/cert-manager-cainjector\" \"uid\":\"5169f4c6-3c35-11ea-baff-eee59f452b94\"] \"spec\":map[\"progressDeadlineSeconds
\":'\\u0258' \"replicas\":'\\x01' \"revisionHistoryLimit\":'\\x02' \"selector\":map[\"matchLabels\":map[\"app\":\"cainje
ctor\" \"release\":\"cert-manager\"]] \"strategy\":map[\"rollingUpdate\":map[\"maxSurge\":\"25%\" \"maxUnavailable\":\"2
5%\"] \"type\":\"RollingUpdate\"] \"template\":map[\"metadata\":map[\"creationTimestamp\":<nil> \"labels\":map[\"app\":\
"cainjector\" \"release\":\"cert-manager\"]] \"spec\":map[\"containers\":[map[\"args\":[\"--v=2\" \"--leader-election-na
mespace=$(POD_NAMESPACE)\"] \"env\":[map[\"name\":\"POD_NAMESPACE\" \"valueFrom\":map[\"fieldRef\":map[\"apiVersion\":\"
v1\" \"fieldPath\":\"metadata.namespace\"]]]] \"image\":\"quay.io/jetstack/cert-manager-cainjector:v0.8.0\" \"imagePullP
olicy\":\"IfNotPresent\" \"name\":\"cainjector\" \"resources\":map[] \"terminationMessagePath\":\"/dev/termination-log\"
 \"terminationMessagePolicy\":\"File\"]] \"dnsPolicy\":\"ClusterFirst\" \"restartPolicy\":\"Always\" \"schedulerName\":\
"default-scheduler\" \"securityContext\":map[] \"serviceAccount\":\"cert-manager-cainjector\" \"serviceAccountName\":\"c
ert-manager-cainjector\" \"terminationGracePeriodSeconds\":'\\x1e']]] \"status\":map[\"availableReplicas\":'\\x01' \"con
ditions\":[map[\"lastTransitionTime\":\"2020-01-21T10:03:48Z\" \"lastUpdateTime\":\"2020-01-21T10:03:51Z\" \"message\":\
"ReplicaSet \\\"cert-manager-cainjector-79b7fc64f\\\" has successfully progressed.\" \"reason\":\"NewReplicaSetAvailable
\" \"status\":\"True\" \"type\":\"Progressing\"] map[\"lastTransitionTime\":\"2020-04-07T14:32:24Z\" \"lastUpdateTime\":
\"2020-04-07T14:32:24Z\" \"message\":\"Deployment has minimum availability.\" \"reason\":\"MinimumReplicasAvailable\" \"
status\":\"True\" \"type\":\"Available\"]] \"observedGeneration\":'\\x01' \"readyReplicas\":'\\x01' \"replicas\":'\\x01'
 \"updatedReplicas\":'\\x01']]}\nfor: \"/tmp/kout370059756\": Deployment.apps \"cert-manager-cainjector\" is invalid: sp
ec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app\":\"cainjector\", \"app.kubernetes.io/c
omponent\":\"cert-manager\", \"app.kubernetes.io/instance\":\"cert-manager-v1.0.0\", \"app.kubernetes.io/managed-by\":\"
kfctl\", \"app.kubernetes.io/name\":\"cert-manager\", \"app.kubernetes.io/part-of\":\"kubeflow\", \"app.kubernetes.io/ve
rsion\":\"v1.0.0\", \"kustomize.component\":\"cert-manager\", \"release\":\"cert-manager\"}, MatchExpressions:[]v1.Label
SelectorRequirement(nil)}: field is immutable, error when creating \"/tmp/kout370059756\": Internal error occurred: fail
ed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request]" filename="kustomi
ze/kustomize.go:202"

Steps to Reproduce the Problem

1. Invoke bundle install on AKS cluster