Installing Istio fails when trying to match resources #762

nimakaviani · 2019-08-09T01:48:23Z

What happened:

I wonder if anyone has tried installing Istio on kind. I get the following errors when trying to install Istio 1.1.11:

deployment.extensions/istio-citadel created
deployment.extensions/istio-sidecar-injector created
horizontalpodautoscaler.autoscaling/istio-ingressgateway created
horizontalpodautoscaler.autoscaling/istio-policy created
horizontalpodautoscaler.autoscaling/istio-telemetry created
horizontalpodautoscaler.autoscaling/istio-pilot created
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector created
unable to recognize "STDIN": no matches for kind "attributemanifest" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "attributemanifest" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "handler" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "handler" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "kubernetes" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"

Looking at this issue knative/serving#2195 it appears to be due to a bug with the the earlier micro versions of K8s 1.11+.

How to reproduce it (as minimally and precisely as possible):

wget -O- https://github.com/istio/istio/releases/download/1.1.11/istio-1.1.11-linux.tar.gz > istio.tgz
tar -zxvf istio.tgz && cd istio-1.1.11
kubectl create namespace istio-system
helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -
helm template install/kubernetes/helm/istio --name istio --namespace istio-system | kubectl apply -f -

Anything else we need to know?:

Environment:

kind version: (use kind version): 0.4.0
Docker version: (use docker info): 18.09
OS (e.g. from /etc/os-release): xenial

The text was updated successfully, but these errors were encountered:

BenTheElder · 2019-08-09T06:10:40Z

kind by default runs Kubernetes v1.15.x at the moment

~~What is your kubectl version?~~ not relevant: see #762 (comment)

Have you looked at https://kind.sigs.k8s.io/docs/user/known-issues/ ?

BenTheElder · 2019-08-09T06:11:49Z

I don't think this is a bug in kind, I'm pretty sure this is an issue with how istio is being installed.

BenTheElder · 2019-08-09T06:13:49Z

Did you skip a step installing CRDs before installing the rest? Those errors relate to those resource types not existing yet. Since they are istio specific a normal Kubernetes cluster generally won't ship them out of the box.

nimakaviani · 2019-08-10T00:25:03Z

No, this snippet installs everything istio needs in any other kubernetes cluster I tried.

wget -O- https://github.com/istio/istio/releases/download/1.1.11/istio-1.1.11-linux.tar.gz > istio.tgz
tar -zxvf istio.tgz && cd istio-1.1.11
kubectl create namespace istio-system
helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -
helm template install/kubernetes/helm/istio --name istio --namespace istio-system | kubectl apply -f -

It is not with Istio only that I see the error, tried Knative too and similar error messages pop up. Only with kind as the cluster.

Here is the command:

kubectl  apply \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving-post-1.14.yaml \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving.yaml \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving-beta-crds.yaml \
   --filename https://github.com/knative/build/releases/download/v0.7.0/build.yaml

similar errors:

unable to recognize "https://github.com/knative/serving/releases/download/v0.7.1/serving-post-1.14.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
unable to recognize "https://github.com/knative/serving/releases/download/v0.7.1/serving.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
unable to recognize "https://github.com/knative/build/releases/download/v0.7.0/build.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
unable to recognize "https://github.com/knative/build/releases/download/v0.7.0/build.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
unable to recognize "https://github.com/knative/build/releases/download/v0.7.0/build.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
unable to recognize "https://github.com/knative/build/releases/download/v0.7.0/build.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"

nimakaviani · 2019-08-10T00:50:22Z

aside from the CRD issue, I wonder if anyone has tried installing Istio on Kind to check whether or not it works.

BenTheElder · 2019-08-10T04:57:49Z

aside from the CRD issue, I wonder if anyone has tried installing Istio on Kind to check whether or not it works.

The istio project has CI running on kind..

BenTheElder · 2019-08-10T04:59:12Z

No, this snippet installs everything istio needs in any other kubernetes cluster I tried

What other clusters did you try?

Have you checked our known issues guide, is your kubectl too far off from the server version?

nimakaviani · 2019-08-12T20:32:52Z

what I noticed is that installing the CRDs fails first time around but if I try to redeploy after a little while it will successfully deploy. Basically similar to this comment knative/serving#2195 (comment) from the issue I linked earlier.

I have tried GKE and IKS with the above installation instructions and I get Istio 1.1.11 without any problem.

my kubectl is at v 1.14 so I doubt there is much regression between the version of the server and the client.

BenTheElder · 2019-08-13T19:26:55Z

This part:

kubectl  apply \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving-post-1.14.yaml \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving.yaml \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving-beta-crds.yaml \
   --filename https://github.com/knative/build/releases/download/v0.7.0/build.yaml

Is both installing CRDs and installing CRs based on them, simultaneously. This is a bad idea, and a race.

BenTheElder · 2019-08-13T19:40:15Z

Installing istio from their latest docs works fine, which involves installing the CRDs seperately first:
https://istio.io/docs/setup/kubernetes/#downloading-the-release
then:
https://istio.io/docs/setup/kubernetes/install/kubernetes/

curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.2.2 sh -
cd istio-1.2.2
for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done
kubectl apply -f install/kubernetes/istio-demo.yaml

BenTheElder · 2019-08-13T19:43:30Z

I then followed this with https://knative.dev/docs/install/knative-with-any-k8s/ which says:

To install Knative, first install the CRDs by running the kubectl apply command once with the -l knative.dev/crd-install=true flag. This prevents race conditions during the install, which cause intermittent errors:

And this worked fine (on the same kind cluster).

This is a problem with their install process. They should ensure that the CRDs are ready to serve before creating CRs from them.

The workaround in their docs does however seem to work.

nimakaviani · 2019-08-13T20:22:01Z

~~Right so the installation instructions on Knative probably need to split the steps for installing the CRs and the CRDs too~~

~~https://knative.dev/docs/install/Knative-with-any-k8s~~

Update: Knative docs actually has the instructions to use the crd selector.

But more importantly, the race issue seems to have been addressed before (see kubernetes/kubernetes#62725 and the corresponding PR), so not sure if this is still a problem in the core of kubernetes. Also following the same install instructions, I dont seem to be able to reproduce this kind of race in any cluster other than Kind.

That's why I wondered maybe Kind has an older version of Kube which like you said does not seem to be the case. so yeah I don't know 🤷‍♂

BenTheElder · 2019-08-13T21:31:10Z

But more importantly, the race issue seems to have been addressed before (see kubernetes/kubernetes#62725 and the corresponding PR), so not sure if this is still a problem in the core of kubernetes.

kubernetes/kubernetes#62725 is about solving the race when you create one shortly after, the problem here is simultaneously creating them.

Think of a CRD (= CustomResourceDefinition) like a table schema in a relational database, and a CR (= CustomResource instance) as an entry in the matching table.

Creating both CRDs and CRs at the same time is then like creating tables while also trying to insert entries into them. Each table needs to be created first, then you can insert.

With the current installation instructions for both Istio and Knative, you do create the CRDs before the CRs, and this works fine.

The old installation instructions were wrong. Trying to create CRs before the CRDs are serving from the API server is a bug in the API usage, it is possible to check that CRDs exist in the API before proceeding (or simply a pause between instructions will generally happen to work).

Also following the same install instructions, I dont seem to be able to reproduce this kind of race in any cluster other than Kind.

On some clusters it may be that you win this race. kind runs on a "noisy shared host" I.E. your development machine with less isolation / dedicated resources.

rjshrjndrn · 2019-08-14T04:17:53Z

@nimakaviani I created a local rancher, and install istio with that. Works fine for me.

nimakaviani · 2019-08-14T06:24:18Z

@rjshrjndrn which installation instructions did you use? The one I referenced or @BenTheElder ‘s or something else?

rjshrjndrn · 2019-08-14T11:40:54Z

@nimakaviani
I didn't tried the helm installation.

Create cluster with Kind
Create rancher container and imported the cluster
From the app catalog, installed istio

BenTheElder · 2019-08-15T23:35:15Z

Going to close this since the current istio / knative docs do not have this issue. If you have more questions please re-open or file a new issue 😅

nimakaviani added the kind/bug Categorizes issue or PR as related to a bug. label Aug 9, 2019

BenTheElder added kind/support Categorizes issue or PR as a support question. triage/needs-information Indicates an issue needs more information in order to work on it. and removed kind/bug Categorizes issue or PR as related to a bug. labels Aug 9, 2019

BenTheElder removed the triage/needs-information Indicates an issue needs more information in order to work on it. label Aug 14, 2019

BenTheElder self-assigned this Aug 15, 2019

BenTheElder closed this as completed Aug 15, 2019

nimakaviani mentioned this issue Sep 8, 2019

Race Condition installing Serving from release yaml knative/serving#2940

Closed

mudler mentioned this issue Mar 31, 2020

kubecf CI failures cloudfoundry-incubator/kubecf#558

Closed

glindsell mentioned this issue May 6, 2020

Kind test script Istio install failing with kubectl version v1.18.x SeldonIO/seldon-core#1802

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installing Istio fails when trying to match resources #762

Installing Istio fails when trying to match resources #762

nimakaviani commented Aug 9, 2019

BenTheElder commented Aug 9, 2019 •

edited

Loading

BenTheElder commented Aug 9, 2019

BenTheElder commented Aug 9, 2019

nimakaviani commented Aug 10, 2019 •

edited

Loading

nimakaviani commented Aug 10, 2019

BenTheElder commented Aug 10, 2019

BenTheElder commented Aug 10, 2019

nimakaviani commented Aug 12, 2019

BenTheElder commented Aug 13, 2019

BenTheElder commented Aug 13, 2019

BenTheElder commented Aug 13, 2019

nimakaviani commented Aug 13, 2019 •

edited

Loading

BenTheElder commented Aug 13, 2019 •

edited

Loading

rjshrjndrn commented Aug 14, 2019

nimakaviani commented Aug 14, 2019 •

edited

Loading

rjshrjndrn commented Aug 14, 2019 •

edited

Loading

BenTheElder commented Aug 15, 2019

Installing Istio fails when trying to match resources #762

Installing Istio fails when trying to match resources #762

Comments

nimakaviani commented Aug 9, 2019

BenTheElder commented Aug 9, 2019 • edited Loading

BenTheElder commented Aug 9, 2019

BenTheElder commented Aug 9, 2019

nimakaviani commented Aug 10, 2019 • edited Loading

nimakaviani commented Aug 10, 2019

BenTheElder commented Aug 10, 2019

BenTheElder commented Aug 10, 2019

nimakaviani commented Aug 12, 2019

BenTheElder commented Aug 13, 2019

BenTheElder commented Aug 13, 2019

BenTheElder commented Aug 13, 2019

nimakaviani commented Aug 13, 2019 • edited Loading

BenTheElder commented Aug 13, 2019 • edited Loading

rjshrjndrn commented Aug 14, 2019

nimakaviani commented Aug 14, 2019 • edited Loading

rjshrjndrn commented Aug 14, 2019 • edited Loading

BenTheElder commented Aug 15, 2019

BenTheElder commented Aug 9, 2019 •

edited

Loading

nimakaviani commented Aug 10, 2019 •

edited

Loading

nimakaviani commented Aug 13, 2019 •

edited

Loading

BenTheElder commented Aug 13, 2019 •

edited

Loading

nimakaviani commented Aug 14, 2019 •

edited

Loading

rjshrjndrn commented Aug 14, 2019 •

edited

Loading