Cannot deploy EFS CSI in cn-north-1 and cn-northwest-1

[binglj]

/kind bug

What happened?
according to the public document below, I should use kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-1.0" to deploy efs csi controller to eks cluster

https://docs.amazonaws.cn/en_us/eks/latest/userguide/efs-csi.html

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?
run kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-1.0" to EKS cluster in China region

Anything else we need to know?:

1. After I ran kubectl apply, and use kubectl describe pod pod-xxx -n kube-system to check the event, I can found

    Normal   Pulling    69s (x3 over 2m17s)  kubelet, ip-192-168-46-97.cn-northwest-1.compute.internal  Pulling image "k8s.gcr.io/sig-storage/csi-provisioner:v2.0.2"
     Warning  Failed     54s (x3 over 2m2s)   kubelet, ip-192-168-46-97.cn-northwest-1.compute.internal  Failed to pull image "k8s.gcr.io/sig-storage/csi-provisioner:v2.0.2": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
     Warning  Failed     54s (x3 over 2m2s)   kubelet, ip-192-168-46-97.cn-northwest-1.compute.internal  Error: ErrImagePull
     Normal   BackOff    30s (x5 over 2m2s)   kubelet, ip-192-168-46-97.cn-northwest-1.compute.internal  Back-off pulling image "k8s.gcr.io/sig-storage/csi-provisioner:v2.0.2"
     Warning  Failed     30s (x5 over 2m2s)   kubelet, ip-192-168-46-97.cn-northwest-1.compute.internal  Error: ImagePullBackOff

Because we cannot access k8s.gcr.io
2. I changed the image url to 048912060910.dkr.ecr.cn-northwest-1.amazonaws.com.cn/gcr/google_containers/sig-storage/csi-provisioner:v2.0.2, then the status of pod changes to CrashLoopBackOff

3. by checking the csi-provisioner logs, I can found,

kubectl logs efs-csi-controller-98666d7f9-mghk8 -c csi-provisioner -n kube-system
I0306 03:25:12.793391       1 csi-provisioner.go:121] Version: v2.0.2
I0306 03:25:12.793450       1 csi-provisioner.go:135] Building kube configs for running in cluster...
I0306 03:25:12.804506       1 connection.go:153] Connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
I0306 03:25:12.804947       1 common.go:111] Probing CSI driver for readiness
I0306 03:25:12.804970       1 connection.go:182] GRPC call: /csi.v1.Identity/Probe
I0306 03:25:12.804980       1 connection.go:183] GRPC request: {}
I0306 03:25:12.807171       1 connection.go:185] GRPC response: {}
I0306 03:25:12.807237       1 connection.go:186] GRPC error: <nil>
I0306 03:25:12.807256       1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo
I0306 03:25:12.807265       1 connection.go:183] GRPC request: {}
I0306 03:25:12.807652       1 connection.go:185] GRPC response: {"name":"efs.csi.aws.com","vendor_version":"v1.1.0-dirty"}
I0306 03:25:12.807732       1 connection.go:186] GRPC error: <nil>
I0306 03:25:12.807749       1 csi-provisioner.go:182] Detected CSI driver efs.csi.aws.com
W0306 03:25:12.807758       1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.
I0306 03:25:12.807773       1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0306 03:25:12.807781       1 connection.go:183] GRPC request: {}
I0306 03:25:12.808200       1 connection.go:185] GRPC response: {}
I0306 03:25:12.808265       1 connection.go:186] GRPC error: <nil>
I0306 03:25:12.808295       1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0306 03:25:12.808306       1 connection.go:183] GRPC request: {}
I0306 03:25:12.808664       1 connection.go:185] GRPC response: {}
I0306 03:25:12.808733       1 connection.go:186] GRPC error: rpc error: code = Unimplemented desc = unknown service csi.v1.Controller
F0306 03:25:12.808757       1 csi-provisioner.go:188] Error getting CSI driver capabilities: rpc error: code = Unimplemented desc = unknown service csi.v1.Controller

Environment

• Kubernetes version (use kubectl version): 1.18
• Driver version: latest

[jumping]

similar error messages.
"
I0310 09:04:01.980196 1 csi-provisioner.go:121] Version: v2.0.2
I0310 09:04:01.980478 1 csi-provisioner.go:135] Building kube configs for running in cluster...
I0310 09:04:01.990872 1 connection.go:153] Connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
I0310 09:04:01.991195 1 common.go:111] Probing CSI driver for readiness
I0310 09:04:01.991207 1 connection.go:182] GRPC call: /csi.v1.Identity/Probe
I0310 09:04:01.991212 1 connection.go:183] GRPC request: {}
I0310 09:04:01.994495 1 connection.go:185] GRPC response: {}
I0310 09:04:01.994538 1 connection.go:186] GRPC error:
I0310 09:04:01.994548 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo
I0310 09:04:01.994595 1 connection.go:183] GRPC request: {}
I0310 09:04:01.994876 1 connection.go:185] GRPC response: {"name":"efs.csi.aws.com","vendor_version":"v1.1.0-dirty"}
I0310 09:04:01.994918 1 connection.go:186] GRPC error:
I0310 09:04:01.994927 1 csi-provisioner.go:182] Detected CSI driver efs.csi.aws.com
W0310 09:04:01.994933 1 metrics.go:142] metrics endpoint will not be started because metrics-address was not specified.
I0310 09:04:01.994945 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0310 09:04:01.994950 1 connection.go:183] GRPC request: {}
I0310 09:04:01.995286 1 connection.go:185] GRPC response: {}
I0310 09:04:01.995319 1 connection.go:186] GRPC error:
I0310 09:04:01.995331 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0310 09:04:01.995338 1 connection.go:183] GRPC request: {}
I0310 09:04:01.995536 1 connection.go:185] GRPC response: {}
I0310 09:04:01.995584 1 connection.go:186] GRPC error: rpc error: code = Unimplemented desc = unknown service csi.v1.Controller
F0310 09:04:01.995597 1 csi-provisioner.go:188] Error getting CSI driver capabilities: rpc error: code = Unimplemented desc = unknown service csi.v1.Controller
"

[sdwerwed]

I think it is related to this #274.

Comment those lines in aws-efs-csi-driver\deploy\kubernetes\overlays\stable\ecr\kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../../../base
images:
#  - name: amazon/aws-efs-csi-driver
#    newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-efs-csi-driver
#    newTag: master
  - name: quay.io/k8scsi/livenessprobe
    newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/livenessprobe
    newTag: v2.0.0
  - name: quay.io/k8scsi/csi-node-driver-registrar
    newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-node-driver-registrar
    newTag: v1.3.0

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.