AWS command error when building AMI #1106

wyattkwancarfax · 2022-11-21T19:21:02Z

What happened:
When building new AMI with v20221112 the build would fail when downloading binaries from s3://amazon-eks.

2022-11-20T23:30:03Z:     amazon-ebs: Downloading binaries from: s3://amazon-eks
2022-11-20T23:30:03Z:     amazon-ebs: AWS cli present - using it to copy binaries from s3.
2022-11-20T23:30:03Z:     amazon-ebs: /tmp/script_8807.sh: line 251: aws: command not found

Line 251 is:
aws s3 cp --region $BINARY_BUCKET_REGION $S3_PATH/$binary

Fix is to re-add awscli back to the scripts/install-worker.sh.
794ed5f#diff-dff0aa3a99c138c812c9364e64a6f95abec4590a27f0fb2b448231342b4a3b89L65

What you expected to happen:
Binaries to be downloaded.

How to reproduce it (as minimally and precisely as possible):
packer build -timestamp-ui -color=false --var aws_region='us-east-1' --var ami_name='amazon-eks-node-1.21-20221121135924' --var binary_bucket_region='us-east-1' --var kubernetes_version='1.21.14' --var kubernetes_build_date='2022-10-31' --var arch='x86_64' --var instance_type='m4.large' eks-worker-al2.json

Anything else we need to know?:
Even though awscli v2 bundle was installed, it couldn't run line 251 when it got to it.
794ed5f#diff-dff0aa3a99c138c812c9364e64a6f95abec4590a27f0fb2b448231342b4a3b89R120-R140

Environment:

AWS Region: us-east-1
Instance Type(s): m4.large
EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.11
Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.21
AMI Version: v20221112
Kernel (e.g. uname -a): Darwin A-W13QNVG40 21.6.0 Darwin Kernel Version 21.6.0: Thu Sep 29 20:13:56 PDT 2022; root:xnu-8020.240.7~1/RELEASE_ARM64_T6000 arm64
Release information (run cat /etc/eks/release on a node):

The text was updated successfully, but these errors were encountered:

cartermckinnon · 2022-11-22T02:41:44Z

Does the addition of --bin-dir /bin/ fix this for you?

https://github.com/awslabs/amazon-eks-ami/blob/master/scripts/install-worker.sh#L138

wyattkwancarfax · 2022-11-22T05:40:12Z

No that didn't work. Still got the same error.

2022-11-22T00:20:20-05:00:     amazon-ebs: Ciphers aes128-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
2022-11-22T00:20:20-05:00:     amazon-ebs: Installing awscli v2 bundle
2022-11-22T00:20:25-05:00:     amazon-ebs: You can now run: /bin//aws --version
2022-11-22T00:20:26-05:00:     amazon-ebs: Loaded plugins: priorities, update-motd, versionlock


...

2022-11-22T00:34:04-05:00:     amazon-ebs: Downloading binaries from: s3://amazon-eks
2022-11-22T00:34:04-05:00:     amazon-ebs: AWS cli present - using it to copy binaries from s3.
2022-11-22T00:34:04-05:00:     amazon-ebs: /tmp/script_9380.sh: line 251: aws: command not found
2022-11-22T00:34:04-05:00: ==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present...
2022-11-22T00:34:04-05:00: ==> amazon-ebs: Terminating the source AWS instance...

cartermckinnon · 2022-11-22T22:32:06Z

Hm, I can't reproduce this. You might want to echo $PATH in install-worker.sh to sanity check it. Does this command include all the variables you're setting in the Packer template?

 packer build \
  -timestamp-ui \
  -color=false \
  --var aws_region='us-east-1' \
  --var ami_name='amazon-eks-node-1.21-20221121135924' \
  --var binary_bucket_region='us-east-1' \
  --var kubernetes_version='1.21.14' \
  --var kubernetes_build_date='2022-10-31' \
  --var arch='x86_64' \
  --var instance_type='m4.large' \
  eks-worker-al2.json

And what version of Packer are you using?

wyattkwancarfax · 2022-11-23T18:35:32Z

We use Jenkins to create our images. Our Jenkins image has packer version v1.7.2 and the $PATH is /usr/local/bin:/usr/bin.

When I run this locally on my mac, my packer version is v1.8.4 and the $PATH is the same.

As for the packer command, that's all the variables we're setting. This was working just last week with v20221104 with this packer command.

packer build \
-timestamp-ui \
-color=false  \
--var aws_region='us-east-1' \ 
--var ami_name='amazon-eks-node-1.21-20221113232535' \   
--var binary_bucket_region='us-east-1'  \
--var kubernetes_version='1.21.2'  \
--var kubernetes_build_date='2021-07-05'  \              
--var arch='x86_64'  \
--var instance_type='m4.large' \        
eks-worker-al2.json

cartermckinnon · 2022-11-30T00:43:10Z

That's the PATH on the packer builder instance, or on your local machines?

wyattkwancarfax · 2022-11-30T14:41:13Z

That's the path on the packer builder instance when I add it to the install-worker.sh file.

cartermckinnon · 2022-12-02T19:52:29Z

Interesting. Can you confirm the user on the packer instance? (throw whoami in install-worker.sh or something?)

You could also try changing the --bin-dir of the AWS CLI installation to /usr/bin to see if that resolves it; that may be a more universal location for various setups.

wyattkwancarfax · 2022-12-05T16:30:49Z

When I add whoami to the script the user is ec2-user.

As for changing --bin-dir to /usr/bin it fails with this message.

2022-12-05T11:23:56-05:00:     amazon-ebs: Installing awscli v2 bundle
2022-12-05T11:23:59-05:00:     amazon-ebs: Got an unexpected argument: /usr/bin
2022-12-05T11:23:59-05:00: ==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present...
2022-12-05T11:23:59-05:00: ==> amazon-ebs: Terminating the source AWS instance...

carlosjgp · 2022-12-05T17:49:32Z

I have the same error with AMIs

amazon-eks-node-1.22-v20221112
amazon-eks-node-1.21-v20221112

Running a script

      {
        "type": "shell",
        "execute_command": "sudo --non-interactive env {{ .Vars }} {{ .Path }}",
        "script": "../../library/base_amazon_linux.sh"
      },

base_amazon_linux.sh

aws s3 cp...

==> amazon-ebs: /tmp/script_276.sh: line 27: aws: command not found

cartermckinnon · 2022-12-05T18:48:45Z

As for changing --bin-dir to /usr/bin

@wyattkwan I meant change the value of the --bin-dir flag to /usr/bin, so you'd have:

sudo "${AWSCLI_DIR}/aws/install" --bin-dir /usr/bin/

This error looks like you removed the flag:

Got an unexpected argument: /usr/bin

cartermckinnon · 2022-12-05T18:56:40Z

@carlosjgp can you check if this change resolves things for you?

wyattkwancarfax · 2022-12-05T19:04:07Z

@cartermckinnon sorry about that. Here is the output once i updated the install-worker.sh file to

sudo "${AWSCLI_DIR}/aws/install" --bin-dir /usr/bin/

2022-12-05T14:00:24-05:00:     amazon-ebs: net.ipv4.ip_forward = 1
2022-12-05T14:00:24-05:00:     amazon-ebs: Downloading binaries from: s3://amazon-eks
2022-12-05T14:00:24-05:00:     amazon-ebs: AWS cli present - using it to copy binaries from s3.
2022-12-05T14:00:24-05:00:     amazon-ebs: /tmp/script_1575.sh: line 253: aws: command not found
2022-12-05T14:00:24-05:00: ==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present...
2022-12-05T14:00:24-05:00: ==> amazon-ebs: Terminating the source AWS instance...

cartermckinnon · 2022-12-05T20:39:04Z

If your PATH includes /usr/bin, and aws is symlinked into /usr/bin, maybe the permission bits on aws are getting messed up? You can replace the usages of aws in install-worker.sh with /usr/bin/aws to see what happens.

Can you add the commit SHA or tag that your checkout is based on so others can try to reproduce? Unless we have a reliable reproduction, I'm not sure what else we can do to root-cause this.

carlosjgp · 2022-12-06T09:48:23Z

@carlosjgp can you check if this change resolves things for you?

I'm building on top of the released AMI. I would need a new release including the PR
#1102

to be able to test this for now I'll install awscli with yum on my pipeline

I'll post an update after building on the next AMI

nwesoccer · 2022-12-20T19:15:42Z

We are experiencing the same issues, is there an ETA on when a new released AMI might get cut with this fix?

cartermckinnon · 2023-01-12T02:35:38Z

We've cut #1102 in the last two releases (v20220305 and v20221222).

We still don't have a reliable repro, so if you can add details @nwesoccer, it could help us get to the bottom of this.

longc · 2023-02-10T11:40:41Z

I applied the following patch "install-worker.sh.patch" to get the aws command not found issue fixed in the pipeline, it looks like there is permission issue for ec2-user to access /usr/local/aws-cli

patch scripts/install-worker.sh < install-worker.sh.patch

--- https://github.com/awslabs/amazon-eks-ami/blob/master/scripts/install-worker.sh.2023-02-03	2023-02-03 00:04:16.000000000 +0000
+++ https://github.com/awslabs/amazon-eks-ami/blob/master/scripts/install-worker.sh.2023-02-09	2023-02-09 16:02:18.000000000 +0000
@@ -126,6 +126,10 @@

 if [[ "$BINARY_BUCKET_REGION" != "us-iso-east-1" && "$BINARY_BUCKET_REGION" != "us-isob-east-1" ]]; then
   # https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
+  ###awscli install patch###
+  echo "Uninstalling awscli v1"
+  sudo yum remove -y awscli
+  ###awscli install patch###
   echo "Installing awscli v2 bundle"
   AWSCLI_DIR=$(mktemp -d)
   curl \
@@ -136,6 +140,9 @@
     -L "https://awscli.amazonaws.com/awscli-exe-linux-${MACHINE}.zip" -o "${AWSCLI_DIR}/awscliv2.zip"
   unzip -q "${AWSCLI_DIR}/awscliv2.zip" -d ${AWSCLI_DIR}
   sudo "${AWSCLI_DIR}/aws/install" --bin-dir /bin/
+  ###awscli install patch###
+  sudo chmod -R 755 /usr/local/aws-cli/
+  ###awscli install patch###
 else
   echo "Installing awscli package"
   sudo yum install -y awscli

wyattkwancarfax · 2023-03-03T20:12:30Z

Thanks for the patch @longc Needed to apply that to get around the aws command not found error.

cartermckinnon closed this as not planned Won't fix, can't repro, duplicate, stale May 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS command error when building AMI #1106

AWS command error when building AMI #1106

wyattkwancarfax commented Nov 21, 2022

cartermckinnon commented Nov 22, 2022

wyattkwancarfax commented Nov 22, 2022

cartermckinnon commented Nov 22, 2022 •

edited

Loading

wyattkwancarfax commented Nov 23, 2022

cartermckinnon commented Nov 30, 2022

wyattkwancarfax commented Nov 30, 2022

cartermckinnon commented Dec 2, 2022 •

edited

Loading

wyattkwancarfax commented Dec 5, 2022

carlosjgp commented Dec 5, 2022

cartermckinnon commented Dec 5, 2022

cartermckinnon commented Dec 5, 2022

wyattkwancarfax commented Dec 5, 2022

cartermckinnon commented Dec 5, 2022

carlosjgp commented Dec 6, 2022

nwesoccer commented Dec 20, 2022

cartermckinnon commented Jan 12, 2023

longc commented Feb 10, 2023 •

edited

Loading

wyattkwancarfax commented Mar 3, 2023

AWS command error when building AMI #1106

AWS command error when building AMI #1106

Comments

wyattkwancarfax commented Nov 21, 2022

cartermckinnon commented Nov 22, 2022

wyattkwancarfax commented Nov 22, 2022

cartermckinnon commented Nov 22, 2022 • edited Loading

wyattkwancarfax commented Nov 23, 2022

cartermckinnon commented Nov 30, 2022

wyattkwancarfax commented Nov 30, 2022

cartermckinnon commented Dec 2, 2022 • edited Loading

wyattkwancarfax commented Dec 5, 2022

carlosjgp commented Dec 5, 2022

cartermckinnon commented Dec 5, 2022

cartermckinnon commented Dec 5, 2022

wyattkwancarfax commented Dec 5, 2022

cartermckinnon commented Dec 5, 2022

carlosjgp commented Dec 6, 2022

nwesoccer commented Dec 20, 2022

cartermckinnon commented Jan 12, 2023

longc commented Feb 10, 2023 • edited Loading

wyattkwancarfax commented Mar 3, 2023

cartermckinnon commented Nov 22, 2022 •

edited

Loading

cartermckinnon commented Dec 2, 2022 •

edited

Loading

longc commented Feb 10, 2023 •

edited

Loading