When replaying user-data in testing will bail user-data when strict due to moving files...

[ajohnstone]

When replaying user-data in testing will bail user-data when strict...

mv: cannot stat ‘/etc/eks/iptables-restore.service’: No such file or directory
Exited with error on line 424

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[suket22]

Thanks for making this change! Did you get a chance to test this out for both Docker and Containerd?

[ajohnstone]

Tested in dockerd, the change is identical for containerd.

[vara-bonthu]

I am hitting the same issue. EKS Managed node groups failed to join the cluster with launch templates and custom user data.

Nodes are healthy but they failed to join the cluster. EC2 system logs shows this error. It doesn't happen all the times. Some node groups works ok.

[ 29.148669] cloud-init[2636]: mv: cannot stat ‘/etc/eks/iptables-restore.service’: No such file or directory

[�[32m  OK  �[0m] Started Kubernetes Kubelet.
[   28.966375] cloud-init[2636]: nvidia-smi not found
[   28.972080] cloud-init[2636]: + B64_CLUSTER_CA=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1EUXhOREV5TlRBMU0xb1hEVE15TURReE1URXlOVEExTTFvd0FIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZFRGtiR3ZFdzJ6OGJ0OUZjTGxSQitwYW8zL2dNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFCUlJpYmttMTM4R0lKeFhra[2022-04-15T10:52:31.291640]zlBLzRyVzl2Sk9JL0Z0UHJsZlg5bTNxZzFGT1hzTTU2Vwp3ejVqUzI5TjVXNktJQ3RpcFJjZ1o4WmNJWUIySEtzOVFucHJZOGdQVEJsWHBwTVZHc0RxSkdhWk5MRzUvRVF0Ckl4VWk3SGJqUUpYcTdBazZRd2FEcjg2eGFSSDh4WU1tcmUvWWhHbnQ2OHBhUmx3cFdXS1JhY3F0V3dvVlZmMFgKQ3c4TTlCR3doaERGSkNlQllFb2RwdURtZDdzcjZSazNqdDhFWmYvQUl4enVpaWIyaGhjdkdERDVkcEFsODh0dApXSnY2UVJQTmd1cmxSZnhOdXpKazNXaXJ4SUtOV2U3RGRQOXdWY1ZhRkFlaW56aTFiV1Z6cGRmd2VYZklTWlhXCjJNMElQa3JsWHB5c09ENnNKdTRjem5xWDdSM2dtRDMyNDdmVgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
[   28.981327] cloud-init[2636]: + API_SERVER_URL=https://F6C3B2AXXXXXXXXX153E5DC9.gr7.eu-west-1.eks.amazonaws.com
[   28.981640] cloud-init[2636]: + K8S_CLUSTER_DNS_IP=172.20.0.10
[   28.982025] cloud-init[2636]: + /etc/eks/bootstrap.sh aws-test-spark-eks --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,eks.amazonaws.com/nodegroup-image=ami-06a0cc2bb4748fc42,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=default-20220415105059649000000002,eks.amazonaws.com/sourceLaunchTemplateId=lt-0d7c107548032e15f --max-pods=29' --b64-cluster-ca LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREGdQVEJsWHBwTVZHc0RxSkdhWk5MRzUvRVF0Ckl4VWk3SGJqUUpYcTdBazZRd2FEcjg2eGFSSDh4WU1tcmUvWWhHbnQ2OHBhUmx3cFdXS1JhY3F0V3dvVlZmMFgKQ3c4TTlCR3doaERGSkNlQllFb2RwdURtZDdzcjZSazNqdDhFWmYvQUl4enVpaWIyaGhjdkdERDVkcEFsODh0dApXSnY2UVJQTmd1cmxSZnhOdXpKazNXaXJ4SUtOV2U3RGRQOXdWY1ZhRkFlaW56aTFiV1Z6cGRmd2VYZklTWlhXCjJNMElQa3JsWHB5c09ENnNKdTRjem5xWDdSM2dtRDMyNDdmVgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== --apiserver-endpoint https://F6C3B2A8A3DXXXXXXXXXX5DC9.gr7.eu-west-1.eks.amazonaws.com --dns-cluster-ip 172.20.0.10 --use-max-pods false
[   29.148669] cloud-init[2636]: mv: cannot stat ‘/etc/eks/iptables-restore.service’: No such file or directory
[   29.151468] cloud-init[2636]: Exited with error on line 485
[   29.152309] cloud-init[2636]: Apr 15 10:52:31 cloud-init[2636]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-002 [1]
[   29.156046] cloud-init[2636]: Apr 15 10:52:31 cloud-init[2636]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
[   29.156851] cloud-init[2636]: Apr 15 10:52:31 cloud-init[2636]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
[   29.170240] cloud-init[2636]: ci-info: no authorized ssh keys fingerprints found for user ec2-user.
ci-info: no authorized ssh keys fingerprints found for user ec2-user.

What's causing this behaviour?

[spkane]

I have run across this as well. I am assuming that this script should ideally be idempotent. I have noticed this error when the bootstrap script is run twice for one reason or another.

[spkane]

@suket22 @ravisinha0506 Any chance that we can get this merged in and get a new AMI built?